[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:09:41 dist-test-slave-zpfg krb5kdc[16790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:09:41 dist-test-slave-zpfg krb5kdc[16790](info): set up 2 sockets
Aug 12 00:09:41 dist-test-slave-zpfg krb5kdc[16790](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 00:09:42.794984 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:09:42 dist-test-slave-zpfg krb5kdc[16790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:09:42.935644 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:41935
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:43.021492 16804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:43.021783 16804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:43.021826 16804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:43.023102 16804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:09:43.023192 16804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:43.023216 16804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:43.023238 16804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:09:43.023279 16804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:09:43.024900 16804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:41935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:43.025216 16804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:43.025482 16804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:09:43.032531 16804 server_base.cc:1034] running on GCE node
W20260812 00:09:43.032624 16811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:43.032572 16810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:43.032776 16813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:43.033083 16804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:43.033499 16804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:43.034881 16804 hybrid_clock.cc:648] HybridClock initialized: now 1786493383034901 us; error 70 us; skew 500 ppm
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:43.038053 16804 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:09:43.038759 16804 webserver.cc:460] Webserver started at http://127.16.99.254:45915/ using document root <none> and password file <none>
I20260812 00:09:43.039027 16804 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:43.039119 16804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:43.039301 16804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:43.040652 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "3478384dbcb0479c874727b8d4c73ef3"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "e797da7685814928fb3f4ad6eeee5092"
server_key_iv: "66c717f3416f93515c87132799249300"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.041095 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "3478384dbcb0479c874727b8d4c73ef3"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "e797da7685814928fb3f4ad6eeee5092"
server_key_iv: "66c717f3416f93515c87132799249300"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.042901 16804 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260812 00:09:43.043959 16820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.044320 16804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:43.044449 16804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "3478384dbcb0479c874727b8d4c73ef3"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "e797da7685814928fb3f4ad6eeee5092"
server_key_iv: "66c717f3416f93515c87132799249300"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.044550 16804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:43.060042 16804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:43.062995 16804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:43.063318 16804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:43.067776 16804 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:41935
I20260812 00:09:43.067826 16874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:41935 every 8 connection(s)
I20260812 00:09:43.068230 16804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:09:43.069388 16875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:43.071910 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3: Bootstrap starting.
I20260812 00:09:43.072290 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 16804
I20260812 00:09:43.072377 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:09:43.072577 16875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:43.072882 16875 log.cc:824] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:43.073076 16783 external_mini_cluster.cc:1428] Setting key cdbdf05cafab6302d11560fcc4c47ab8
I20260812 00:09:43.073661 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3: No bootstrap required, opened a new log
I20260812 00:09:43.076205 16875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3478384dbcb0479c874727b8d4c73ef3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41935 } }
I20260812 00:09:43.076475 16875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:43.076541 16875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3478384dbcb0479c874727b8d4c73ef3, State: Initialized, Role: FOLLOWER
I20260812 00:09:43.076668 16875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3478384dbcb0479c874727b8d4c73ef3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41935 } }
I20260812 00:09:43.076768 16875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:09:43.076804 16875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:09:43.076846 16875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:43.077888 16875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3478384dbcb0479c874727b8d4c73ef3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41935 } }
I20260812 00:09:43.077983 16875 leader_election.cc:302] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3478384dbcb0479c874727b8d4c73ef3; no voters:
I20260812 00:09:43.078197 16875 leader_election.cc:288] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:09:43.078227 16880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:43.078366 16880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 3478384dbcb0479c874727b8d4c73ef3, State: Running, Role: LEADER
I20260812 00:09:43.078462 16880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3478384dbcb0479c874727b8d4c73ef3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41935 } }
I20260812 00:09:43.078495 16875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:09:43.080778 16881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3478384dbcb0479c874727b8d4c73ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3478384dbcb0479c874727b8d4c73ef3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41935 } } }
I20260812 00:09:43.080888 16881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:43.080750 16882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3478384dbcb0479c874727b8d4c73ef3. Latest consensus state: current_term: 1 leader_uuid: "3478384dbcb0479c874727b8d4c73ef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3478384dbcb0479c874727b8d4c73ef3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41935 } } }
I20260812 00:09:43.081446 16882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3 [sys.catalog]: This master's current role is: LEADER
W20260812 00:09:43.081669 16895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:09:43.081736 16895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:09:43.081796 16898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:09:43.082015 16898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:09:43.083930 16898 catalog_manager.cc:1347] Generated new cluster ID: 4b104f805fa74993b4f20ada24609634
I20260812 00:09:43.083993 16898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:43.092387 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.074024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.16.99.254:41935)
0812 00:09:43.074271 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:43.074281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:43.074536 (+ 255us) server_negotiation.cc:407] Connection header received
0812 00:09:43.077084 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:43.077091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:43.077148 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:43.077299 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:43.079466 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.081266 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.082693 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.083166 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.083304 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.087539 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:43.087543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:43.087544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:43.087555 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:43.090170 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.090725 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.090727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.090728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.090832 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.091145 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.091146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.091147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.091432 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:43.091625 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:43.091942 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 00:09:43.092106 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":105,"threads_started":1}
I20260812 00:09:43.103394 16898 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:09:43.104027 16898 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:09:43.109439 16898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3478384dbcb0479c874727b8d4c73ef3: Generated new TSK 0
I20260812 00:09:43.109700 16898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:09:43.149653 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:41935
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:43.242321 16903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:43.242620 16903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:43.242650 16903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:43.243921 16903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:43.243978 16903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:43.244007 16903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:09:43.245661 16903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:41935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:43.245905 16903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:43.246157 16903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:43.251665 16914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:43.251693 16911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:43.251798 16908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:43.252099 16903 server_base.cc:1034] running on GCE node
I20260812 00:09:43.252429 16903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:43.252728 16903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:43.253906 16903 hybrid_clock.cc:648] HybridClock initialized: now 1786493383253877 us; error 44 us; skew 500 ppm
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:43.256951 16903 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:09:43.257475 16903 webserver.cc:460] Webserver started at http://127.16.99.193:37185/ using document root <none> and password file <none>
I20260812 00:09:43.257745 16903 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:43.257824 16903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:43.257982 16903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:43.259286 16903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "6aefad8ea85143739c3a41908f13206a"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "ee05734d475652a9e3f05a94d44664a6"
server_key_iv: "484dcbbe2f48c38e5fdad8e75e6e700d"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.259670 16903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "6aefad8ea85143739c3a41908f13206a"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "ee05734d475652a9e3f05a94d44664a6"
server_key_iv: "484dcbbe2f48c38e5fdad8e75e6e700d"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.261169 16903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:09:43.262229 16921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.262524 16903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:43.262645 16903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "6aefad8ea85143739c3a41908f13206a"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "ee05734d475652a9e3f05a94d44664a6"
server_key_iv: "484dcbbe2f48c38e5fdad8e75e6e700d"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.262728 16903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:43.273541 16903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:43.276134 16903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:43.276376 16903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:43.276705 16903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:43.277056 16903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:43.277115 16903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.277153 16903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:43.277181 16903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.283998 16903 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:40003
I20260812 00:09:43.284068 17043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:40003 every 8 connection(s)
I20260812 00:09:43.284473 16903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:09:43.286465 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 16903
I20260812 00:09:43.286631 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:09:43.286821 16783 external_mini_cluster.cc:1428] Setting key c42f59676d7c7883c9da70befe6c4e8c
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
I20260812 00:09:43.297950 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.285614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:40359 (local address 127.16.99.254:41935)
0812 00:09:43.285698 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:43.285701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:43.286035 (+ 334us) server_negotiation.cc:407] Connection header received
0812 00:09:43.287023 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:43.287025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:43.287040 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:43.287240 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:43.289192 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.289830 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.291042 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.291259 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.291339 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.294260 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:43.294262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:43.294262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:43.294272 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:43.296212 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.296756 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.296757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.296758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.296820 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.297125 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.297126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.297126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.297264 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:43.297344 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:43.297761 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 00:09:43.297867 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:09:43.298203 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.285567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:41935 (local address 127.16.99.193:40359)
0812 00:09:43.285861 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:43.285925 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:43.286866 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:43.287398 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:43.287405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:43.288616 (+ 1211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:43.288998 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:43.289005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.289950 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.289951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:43.290929 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:43.290932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.291349 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.291350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:43.291493 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.292118 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:43.292130 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:43.294022 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:43.296333 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:43.296346 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:43.296348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:43.296652 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:43.296899 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:43.296899 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:43.296900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:43.297043 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:43.297344 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:43.297353 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:43.297547 (+ 194us) client_negotiation.cc:769] Sending connection context
0812 00:09:43.297708 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 00:09:43.297908 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
I20260812 00:09:43.298856 17044 heartbeater.cc:347] Connected to a master server at 127.16.99.254:41935
I20260812 00:09:43.298995 17044 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:43.299224 17044 heartbeater.cc:511] Master 127.16.99.254:41935 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
I20260812 00:09:43.299908 16837 ts_manager.cc:194] Registered new tserver with Master: 6aefad8ea85143739c3a41908f13206a (127.16.99.193:40003)
I20260812 00:09:43.300663 16837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:40359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:09:43.319164 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:41935
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:43.406203 17051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:43.406459 17051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:43.406486 17051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:43.407757 17051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:43.407812 17051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:43.407841 17051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:09:43.409382 17051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:41935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:43.409636 17051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:43.409898 17051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:43.415239 17060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:43.415304 17057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:43.415246 17058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:43.415818 17051 server_base.cc:1034] running on GCE node
I20260812 00:09:43.416229 17051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:43.416538 17051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:43.417747 17051 hybrid_clock.cc:648] HybridClock initialized: now 1786493383417746 us; error 46 us; skew 500 ppm
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:43.420873 17051 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:09:43.421320 17051 webserver.cc:460] Webserver started at http://127.16.99.194:34791/ using document root <none> and password file <none>
I20260812 00:09:43.421514 17051 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:43.421556 17051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:43.421654 17051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:43.422863 17051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "d1c8215e729f4841a4d4c9029393c650"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "db83963ce36001ab9a2ca90375bc3592"
server_key_iv: "09aab30e44dbbd32caf989c429440aa9"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.423208 17051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "d1c8215e729f4841a4d4c9029393c650"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "db83963ce36001ab9a2ca90375bc3592"
server_key_iv: "09aab30e44dbbd32caf989c429440aa9"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.424659 17051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:43.425606 17067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.425931 17051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:43.426023 17051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "d1c8215e729f4841a4d4c9029393c650"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "db83963ce36001ab9a2ca90375bc3592"
server_key_iv: "09aab30e44dbbd32caf989c429440aa9"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.426084 17051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:43.451141 17051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:43.453649 17051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:43.453783 17051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:43.453994 17051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:43.454404 17051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:43.454450 17051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.454473 17051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:43.454486 17051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.460702 17051 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:42995
I20260812 00:09:43.460803 17191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:42995 every 8 connection(s)
I20260812 00:09:43.461151 17051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:09:43.465685 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 17051
I20260812 00:09:43.465873 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:09:43.466130 16783 external_mini_cluster.cc:1428] Setting key f1a9bc16c94a2b81b00683295f961fb8
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:43.474261 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.462095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:43769 (local address 127.16.99.254:41935)
0812 00:09:43.462193 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:43.462196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:43.462405 (+ 209us) server_negotiation.cc:407] Connection header received
0812 00:09:43.463036 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:43.463038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:43.463049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:43.463120 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:43.464485 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.465053 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.466878 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.467192 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.467261 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.470457 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:43.470460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:43.470460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:43.470470 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:43.472310 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.473007 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.473009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.473009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.473082 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.473437 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.473438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.473439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.473621 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:43.473700 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:43.474010 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 00:09:43.474117 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 00:09:43.474305 17194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.462042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:41935 (local address 127.16.99.194:43769)
0812 00:09:43.462233 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:43.462283 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:43.462920 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:43.463357 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:43.463368 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:43.464041 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:43.464280 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:43.464285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.465202 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.465202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:43.466776 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:43.466779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.467301 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.467302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:43.467486 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.468128 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:43.468140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:43.470272 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:43.472417 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:43.472426 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:43.472429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:43.472894 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:43.473190 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:43.473192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:43.473192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:43.473341 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:43.473696 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:43.473701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:43.473850 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 00:09:43.473938 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 00:09:43.474085 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260812 00:09:43.474918 17192 heartbeater.cc:347] Connected to a master server at 127.16.99.254:41935
I20260812 00:09:43.475036 17192 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:43.475270 17192 heartbeater.cc:511] Master 127.16.99.254:41935 requested a full tablet report, sending...
I20260812 00:09:43.475759 16837 ts_manager.cc:194] Registered new tserver with Master: d1c8215e729f4841a4d4c9029393c650 (127.16.99.194:42995)
I20260812 00:09:43.476184 16837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:43769
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:09:43.501979 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:41935
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:43.593182 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:43.593484 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:43.593530 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:43.594864 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:43.594949 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:43.595000 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:09:43.597013 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:41935
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:43.597353 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:43.597640 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:43.603188 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:43.603322 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:43.603222 17197 server_base.cc:1034] running on GCE node
W20260812 00:09:43.603207 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:43.603631 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:43.603859 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:43.605018 17197 hybrid_clock.cc:648] HybridClock initialized: now 1786493383604995 us; error 79 us; skew 500 ppm
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:43.607923 17197 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:09:43.608439 17197 webserver.cc:460] Webserver started at http://127.16.99.195:33111/ using document root <none> and password file <none>
I20260812 00:09:43.608635 17197 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:43.608675 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:43.608767 17197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:43.609818 17197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "2b70f905def64ae3b5a871034fec9aa0"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "9b46107c603b0e60491b219fa4b4d25c"
server_key_iv: "da391fcac1e5684fddeab8579095a199"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.610191 17197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "2b70f905def64ae3b5a871034fec9aa0"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "9b46107c603b0e60491b219fa4b4d25c"
server_key_iv: "da391fcac1e5684fddeab8579095a199"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.611753 17197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:43.612660 17213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.612906 17197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:43.612987 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "2b70f905def64ae3b5a871034fec9aa0"
format_stamp: "Formatted at 2026-08-12 00:09:43 on dist-test-slave-zpfg"
server_key: "9b46107c603b0e60491b219fa4b4d25c"
server_key_iv: "da391fcac1e5684fddeab8579095a199"
server_key_version: "encryptionkey@0"
I20260812 00:09:43.613080 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:43.623086 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:43.625660 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:43.625794 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:43.626008 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:43.626399 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:43.626441 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.626466 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:43.626510 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:43.633426 17197 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:38387
I20260812 00:09:43.633476 17335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:38387 every 8 connection(s)
I20260812 00:09:43.633867 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:09:43.638573 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 17197
I20260812 00:09:43.638684 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:09:43.638845 16783 external_mini_cluster.cc:1428] Setting key b16c3a564a11244a63310bb58e9ef876
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:43.644968 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.634877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:42079 (local address 127.16.99.254:41935)
0812 00:09:43.635006 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:43.635008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:43.635281 (+ 273us) server_negotiation.cc:407] Connection header received
0812 00:09:43.635857 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:43.635859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:43.635872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:43.635925 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:43.637027 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.637550 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.638672 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.638939 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.639006 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.641814 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:43.641816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:43.641816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:43.641825 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:43.643310 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.643866 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.643868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.643868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.643937 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.644226 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.644227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.644228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.644361 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:43.644456 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:43.644767 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 00:09:43.644868 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 00:09:43.645200 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.634785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:41935 (local address 127.16.99.195:42079)
0812 00:09:43.635037 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:43.635172 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:43.635733 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:43.636017 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:43.636020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:43.636680 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:43.636896 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:43.636902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.637657 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.637659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:43.638579 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:43.638582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.639016 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.639017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:43.639232 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.639945 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:43.639954 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:43.641683 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:43.643432 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:43.643440 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:43.643443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:43.643776 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:43.644011 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:43.644012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:43.644012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:43.644139 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:43.644449 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:43.644457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:43.644620 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 00:09:43.644742 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 00:09:43.644927 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260812 00:09:43.645897 17336 heartbeater.cc:347] Connected to a master server at 127.16.99.254:41935
I20260812 00:09:43.646035 17336 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:43.646268 17336 heartbeater.cc:511] Master 127.16.99.254:41935 requested a full tablet report, sending...
I20260812 00:09:43.646766 16837 ts_manager.cc:194] Registered new tserver with Master: 2b70f905def64ae3b5a871034fec9aa0 (127.16.99.195:38387)
I20260812 00:09:43.647271 16837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:42079
I20260812 00:09:43.651386 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:09:43.679023 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.670450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.16.99.193:40003)
0812 00:09:43.670671 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:43.670675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:43.670687 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:09:43.670715 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:43.670716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:43.670731 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:43.670806 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:43.671491 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.672389 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.673201 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.673640 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.673708 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.676136 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:43.676146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:43.676146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:43.676154 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:43.677749 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.678102 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.678104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.678105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.678162 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.678453 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.678454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.678454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.678609 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:43.678730 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:43.678824 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 00:09:43.678892 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
W20260812 00:09:43.679615 16951 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59690
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[16790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:09:43.705912 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:43.697678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.16.99.193:40003)
0812 00:09:43.697817 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:43.697819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:43.697945 (+ 126us) server_negotiation.cc:407] Connection header received
0812 00:09:43.698064 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:43.698065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:43.698086 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:43.698156 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:43.698871 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.699426 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.700232 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:43.700445 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:43.700532 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:43.703279 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:43.703281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:43.703281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:43.703291 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:43.704661 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.705073 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.705075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.705075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.705137 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:43.705347 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:43.705348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:43.705349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:43.705463 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:43.705590 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:43.705714 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 00:09:43.705800 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260812 00:09:43.706604 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 16903
I20260812 00:09:43.712644 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 17051
I20260812 00:09:43.718859 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 17197
I20260812 00:09:43.724071 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 16804
2026-08-12T00:09:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2125 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[17350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[17350](info): set up 2 sockets
Aug 12 00:09:43 dist-test-slave-zpfg krb5kdc[17350](info): commencing operation
krb5kdc: starting...
W20260812 00:09:44.804986 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:09:44 dist-test-slave-zpfg krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:09:44.945842 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33997
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:33997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:45.031572 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:45.031862 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:45.031906 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:45.033185 17366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:09:45.033262 17366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:45.033285 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:45.033308 17366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:09:45.033339 17366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:09:45.035017 17366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:33997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:45.035341 17366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:45.035585 17366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:45.041726 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:45.041893 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:45.041911 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:45.042706 17366 server_base.cc:1034] running on GCE node
I20260812 00:09:45.042954 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:45.043263 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:45.044461 17366 hybrid_clock.cc:648] HybridClock initialized: now 1786493385044453 us; error 38 us; skew 500 ppm
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:45.047499 17366 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:09:45.048043 17366 webserver.cc:460] Webserver started at http://127.16.99.254:33657/ using document root <none> and password file <none>
I20260812 00:09:45.048291 17366 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:45.048363 17366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:45.048518 17366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:45.049670 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "a57d175469e54f69bf53b7d6f245221c"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "1941cd7a7b37f0061590922e43ac7dfc"
server_key_iv: "63dd9783145646b4cf265e2193249bfd"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.050048 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "a57d175469e54f69bf53b7d6f245221c"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "1941cd7a7b37f0061590922e43ac7dfc"
server_key_iv: "63dd9783145646b4cf265e2193249bfd"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.051656 17366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:09:45.052587 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.052829 17366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:45.052959 17366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "a57d175469e54f69bf53b7d6f245221c"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "1941cd7a7b37f0061590922e43ac7dfc"
server_key_iv: "63dd9783145646b4cf265e2193249bfd"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.053052 17366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:45.076979 17366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:45.079770 17366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:45.079977 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:45.084856 17366 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:33997
I20260812 00:09:45.084894 17439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:33997 every 8 connection(s)
I20260812 00:09:45.085347 17366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:09:45.086374 17441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:45.088792 17441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c: Bootstrap starting.
I20260812 00:09:45.089351 17441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:45.089594 17441 log.cc:824] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:45.090322 17441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c: No bootstrap required, opened a new log
I20260812 00:09:45.092020 17441 raft_consensus.cc:348] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57d175469e54f69bf53b7d6f245221c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33997 } }
I20260812 00:09:45.092170 17441 raft_consensus.cc:374] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:45.092192 17441 raft_consensus.cc:729] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57d175469e54f69bf53b7d6f245221c, State: Initialized, Role: FOLLOWER
I20260812 00:09:45.092340 17441 consensus_queue.cc:260] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57d175469e54f69bf53b7d6f245221c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33997 } }
I20260812 00:09:45.092424 17441 raft_consensus.cc:388] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:09:45.092478 17441 raft_consensus.cc:482] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:09:45.092530 17441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:45.092655 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 17366
I20260812 00:09:45.092749 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:09:45.092933 16783 external_mini_cluster.cc:1428] Setting key 336be750511dda2c3fbab804698657d6
I20260812 00:09:45.093390 17441 raft_consensus.cc:504] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57d175469e54f69bf53b7d6f245221c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33997 } }
I20260812 00:09:45.093509 17441 leader_election.cc:302] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a57d175469e54f69bf53b7d6f245221c; no voters:
I20260812 00:09:45.093714 17441 leader_election.cc:288] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:09:45.093736 17446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:45.093956 17446 raft_consensus.cc:686] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [term 1 LEADER]: Becoming Leader. State: Replica: a57d175469e54f69bf53b7d6f245221c, State: Running, Role: LEADER
I20260812 00:09:45.093984 17441 sys_catalog.cc:564] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:09:45.094053 17446 consensus_queue.cc:237] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57d175469e54f69bf53b7d6f245221c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33997 } }
I20260812 00:09:45.094605 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a57d175469e54f69bf53b7d6f245221c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57d175469e54f69bf53b7d6f245221c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33997 } } }
I20260812 00:09:45.094697 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:45.094874 17448 sys_catalog.cc:455] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a57d175469e54f69bf53b7d6f245221c. Latest consensus state: current_term: 1 leader_uuid: "a57d175469e54f69bf53b7d6f245221c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a57d175469e54f69bf53b7d6f245221c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33997 } } }
I20260812 00:09:45.094957 17448 sys_catalog.cc:458] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:45.095477 17455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:09:45.096122 17455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:09:45.097795 17455 catalog_manager.cc:1347] Generated new cluster ID: c1dc13a6327d47bc9a76aebf4445e4dc
I20260812 00:09:45.097856 17455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:45.104861 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.093514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.16.99.254:33997)
0812 00:09:45.093734 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.093745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.093832 (+ 87us) server_negotiation.cc:407] Connection header received
0812 00:09:45.094168 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.094179 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.094245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.095374 (+ 1129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:45.095534 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.096655 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.097585 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.098151 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.098284 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.100775 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:45.100784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:45.100784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:45.100796 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:45.102921 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.103357 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.103361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.103361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.103481 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.103766 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.103767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.103768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.104076 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:45.104369 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.104455 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.104635 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
I20260812 00:09:45.114256 17455 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:09:45.115064 17455 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:09:45.123843 17455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a57d175469e54f69bf53b7d6f245221c: Generated new TSK 0
I20260812 00:09:45.124085 17455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:09:45.169780 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33997
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:45.263345 17468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:45.263585 17468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:45.263612 17468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:45.264878 17468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:45.264932 17468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:45.264961 17468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:09:45.266755 17468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:45.267040 17468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:45.267318 17468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:45.273384 17474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:45.273480 17473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:45.273409 17476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:45.273583 17468 server_base.cc:1034] running on GCE node
I20260812 00:09:45.273808 17468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:45.274092 17468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:45.275303 17468 hybrid_clock.cc:648] HybridClock initialized: now 1786493385275274 us; error 49 us; skew 500 ppm
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:45.278141 17468 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:09:45.278748 17468 webserver.cc:460] Webserver started at http://127.16.99.193:37301/ using document root <none> and password file <none>
I20260812 00:09:45.278992 17468 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:45.279042 17468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:45.279193 17468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:45.280339 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "7d008a9cde314c02b56a3629e45c843b"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "96b8704261703a87797259b24d3d6f3c"
server_key_iv: "5d86bdf6cecf0a717f19d4521c8ddb1c"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.280823 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "7d008a9cde314c02b56a3629e45c843b"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "96b8704261703a87797259b24d3d6f3c"
server_key_iv: "5d86bdf6cecf0a717f19d4521c8ddb1c"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.282765 17468 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260812 00:09:45.283792 17483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.284026 17468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:45.284233 17468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "7d008a9cde314c02b56a3629e45c843b"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "96b8704261703a87797259b24d3d6f3c"
server_key_iv: "5d86bdf6cecf0a717f19d4521c8ddb1c"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.284335 17468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:45.300722 17468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:45.303686 17468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:45.303890 17468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:45.304194 17468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:45.304570 17468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:45.304642 17468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.304682 17468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:45.304735 17468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.312099 17468 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:45227
I20260812 00:09:45.312163 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:45227 every 8 connection(s)
I20260812 00:09:45.312574 17468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:09:45.317155 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 17468
I20260812 00:09:45.317322 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:09:45.317495 16783 external_mini_cluster.cc:1428] Setting key bc925a684b5a10ad5358739867174516
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:45.324831 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.313581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:39329 (local address 127.16.99.254:33997)
0812 00:09:45.313688 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.313690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.313876 (+ 186us) server_negotiation.cc:407] Connection header received
0812 00:09:45.315010 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.315012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.315029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.315100 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:45.315911 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.316549 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.317980 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.318249 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.318318 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.320945 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:45.320947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:45.320947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:45.320958 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:45.323199 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.323747 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.323749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.323749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.323810 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.324123 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.324124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.324124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.324260 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:45.324342 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.324625 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.324734 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 00:09:45.325042 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.313494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33997 (local address 127.16.99.193:39329)
0812 00:09:45.313731 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.313777 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.314852 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.315227 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.315231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.315526 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:45.315799 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.315804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.316693 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.316694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.317888 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.317891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.318328 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.318329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.318512 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.319121 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:45.319131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:45.320827 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:45.323308 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:45.323317 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:45.323319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:45.323582 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:45.323903 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:45.323904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:45.323904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:45.324048 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:45.324332 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:45.324340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:45.324486 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.324601 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.324780 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260812 00:09:45.325704 17612 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33997
I20260812 00:09:45.325834 17612 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:45.326056 17612 heartbeater.cc:511] Master 127.16.99.254:33997 requested a full tablet report, sending...
I20260812 00:09:45.326766 17402 ts_manager.cc:194] Registered new tserver with Master: 7d008a9cde314c02b56a3629e45c843b (127.16.99.193:45227)
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
I20260812 00:09:45.327553 17402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:39329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:09:45.349841 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33997
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:45.442535 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:45.442752 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:45.442780 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:45.443953 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:45.443987 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:45.444013 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:09:45.445639 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:45.445919 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:45.446117 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:45.452418 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:45.452428 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:45.452471 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:45.452469 17617 server_base.cc:1034] running on GCE node
I20260812 00:09:45.452955 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:45.453195 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:45.454406 17617 hybrid_clock.cc:648] HybridClock initialized: now 1786493385454368 us; error 53 us; skew 500 ppm
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:45.457435 17617 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:09:45.457926 17617 webserver.cc:460] Webserver started at http://127.16.99.194:38279/ using document root <none> and password file <none>
I20260812 00:09:45.458204 17617 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:45.458278 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:45.458457 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:45.459657 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "40c241a47eb44f80a2e5d648d8f479cb"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "6af32bd19c752a2bf3d45dd44664f310"
server_key_iv: "ab3b5d1d45603d402735e4142d0c9090"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.460036 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "40c241a47eb44f80a2e5d648d8f479cb"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "6af32bd19c752a2bf3d45dd44664f310"
server_key_iv: "ab3b5d1d45603d402735e4142d0c9090"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.461496 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:09:45.462631 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.462931 17617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:45.463056 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "40c241a47eb44f80a2e5d648d8f479cb"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "6af32bd19c752a2bf3d45dd44664f310"
server_key_iv: "ab3b5d1d45603d402735e4142d0c9090"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.463145 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:45.473356 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:45.476378 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:45.476572 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:45.476850 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:45.477198 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:45.477255 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.477289 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:45.477319 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.483641 17617 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:35007
I20260812 00:09:45.483723 17758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:35007 every 8 connection(s)
I20260812 00:09:45.484092 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:09:45.486694 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 17617
I20260812 00:09:45.486805 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:09:45.486981 16783 external_mini_cluster.cc:1428] Setting key 40d901fbb65f0001d9fe77fe6c4ed93a
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:45.497429 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.484989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:49889 (local address 127.16.99.254:33997)
0812 00:09:45.485085 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.485087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.485282 (+ 195us) server_negotiation.cc:407] Connection header received
0812 00:09:45.486507 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.486508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.486522 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.486591 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:45.487696 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.488400 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.490118 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.490510 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.490649 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.493475 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:45.493477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:45.493477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:45.493488 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:45.495720 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.496241 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.496242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.496243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.496322 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.496624 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.496625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.496625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.496800 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:45.496870 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.497180 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.497286 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 00:09:45.497504 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.484940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33997 (local address 127.16.99.194:49889)
0812 00:09:45.485111 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.485156 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.486099 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.486814 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.486820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.487231 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:45.487518 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.487524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.488499 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.488501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.490017 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.490020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.490607 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.490608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.490779 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.491504 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:45.491513 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:45.493357 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:45.495837 (+ 2480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:45.495846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:45.495848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:45.496141 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:45.496420 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:45.496421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:45.496421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:45.496544 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:45.496890 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:45.496896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:45.497033 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.497127 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.497292 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260812 00:09:45.498158 17759 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33997
I20260812 00:09:45.498262 17759 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
I20260812 00:09:45.498515 17759 heartbeater.cc:511] Master 127.16.99.254:33997 requested a full tablet report, sending...
I20260812 00:09:45.498932 17402 ts_manager.cc:194] Registered new tserver with Master: 40c241a47eb44f80a2e5d648d8f479cb (127.16.99.194:35007)
I20260812 00:09:45.499317 17402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:49889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:09:45.521811 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33997
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:45.608821 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:45.609042 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:45.609067 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:45.610260 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:45.610304 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:45.610332 17764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:09:45.611981 17764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33997
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:45.612248 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:45.612493 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:45.617820 17769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:45.617888 17772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:45.618022 17764 server_base.cc:1034] running on GCE node
W20260812 00:09:45.618269 17770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:45.618700 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:45.618989 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:45.620186 17764 hybrid_clock.cc:648] HybridClock initialized: now 1786493385620158 us; error 41 us; skew 500 ppm
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:45.627671 17764 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:09:45.628327 17764 webserver.cc:460] Webserver started at http://127.16.99.195:40333/ using document root <none> and password file <none>
I20260812 00:09:45.628579 17764 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:45.628628 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:45.628768 17764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:45.629870 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "e04de1df424b4d6aa762e58014a0f51f"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "b9e1866daaa6ce5c95afd88c4e998e92"
server_key_iv: "d423cbedca9a73446f6efa8cef2d045a"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.630267 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "e04de1df424b4d6aa762e58014a0f51f"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "b9e1866daaa6ce5c95afd88c4e998e92"
server_key_iv: "d423cbedca9a73446f6efa8cef2d045a"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.632408 17764 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260812 00:09:45.633430 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.633654 17764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:45.633774 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "e04de1df424b4d6aa762e58014a0f51f"
format_stamp: "Formatted at 2026-08-12 00:09:45 on dist-test-slave-zpfg"
server_key: "b9e1866daaa6ce5c95afd88c4e998e92"
server_key_iv: "d423cbedca9a73446f6efa8cef2d045a"
server_key_version: "encryptionkey@0"
I20260812 00:09:45.633845 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:45.651466 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:45.654258 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:45.654479 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:45.654731 17764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:45.655028 17764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:45.655064 17764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.655089 17764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:45.655103 17764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:45.662257 17764 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:36797
I20260812 00:09:45.662467 17905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:36797 every 8 connection(s)
I20260812 00:09:45.662788 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:09:45.668273 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 17764
I20260812 00:09:45.668406 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:09:45.668579 16783 external_mini_cluster.cc:1428] Setting key 93cbac47808ce476bf85f2a664b3a4b8
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:45.674672 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.663743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40071 (local address 127.16.99.254:33997)
0812 00:09:45.663819 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.663822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.664040 (+ 218us) server_negotiation.cc:407] Connection header received
0812 00:09:45.665220 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.665222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.665236 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.665305 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:45.666068 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.666726 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.667959 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.668207 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.668292 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.671340 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:45.671343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:45.671343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:45.671355 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:45.672875 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.673487 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.673488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.673489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.673561 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.673879 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.673880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.673880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.674015 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:45.674103 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.674456 (+ 353us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.674555 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:09:45.674944 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.663689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33997 (local address 127.16.99.195:40071)
0812 00:09:45.663896 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.663937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.665079 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.665398 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.665403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.665730 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:45.665952 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.665957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.666838 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.666840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.667849 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.667851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.668320 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.668321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.668501 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.669256 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:45.669269 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:45.671215 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:45.673003 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:45.673013 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:45.673016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:45.673387 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:45.673655 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:45.673656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:45.673657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:45.673794 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:45.674092 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:45.674108 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:45.674267 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.674430 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.674629 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260812 00:09:45.675680 17906 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33997
I20260812 00:09:45.675809 17906 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:45.676024 17906 heartbeater.cc:511] Master 127.16.99.254:33997 requested a full tablet report, sending...
I20260812 00:09:45.676576 17402 ts_manager.cc:194] Registered new tserver with Master: e04de1df424b4d6aa762e58014a0f51f (127.16.99.195:36797)
I20260812 00:09:45.677037 17402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:40071
I20260812 00:09:45.680806 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:09:45.689718 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.682614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.16.99.254:33997)
0812 00:09:45.682678 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.682680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.682847 (+ 167us) server_negotiation.cc:407] Connection header received
0812 00:09:45.682876 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.682876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.682887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.682958 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:45.683627 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.684251 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.685099 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.685302 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.685382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.686915 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:45.686917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:45.686917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:45.686924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:45.688390 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.688829 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.688830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.688831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.688887 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.689140 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.689141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.689141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.689385 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:45.689465 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.689579 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.689644 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:09:45.692560 17402 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:09:45.694423 17402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:45.701763 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.698003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.16.99.195:36797)
0812 00:09:45.698197 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.698202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.698212 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:09:45.698252 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.698254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.698270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.698394 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:45.698507 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.699282 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.700746 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.701242 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.701333 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.701393 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.701609 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.701660 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260812 00:09:45.701769 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.697796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:36797 (local address 127.0.0.1:50468)
0812 00:09:45.698016 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.698027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.698077 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.698392 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.698393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.698395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:45.698449 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.698450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.699395 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.699396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.700679 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.700683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.701333 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.701334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.701418 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.701433 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.701581 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.701646 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":34,"thread_start_us":73,"threads_started":1}
I20260812 00:09:45.702171 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.697537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45227 (local address 127.0.0.1:43466)
0812 00:09:45.697726 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.697764 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.697825 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.698155 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.698159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.698161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:45.698399 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.698402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.699344 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.699345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.700374 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.700377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.701897 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.701899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.701975 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.701978 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.702034 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.702080 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":20,"thread_start_us":155,"threads_started":1}
I20260812 00:09:45.702183 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.697695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.16.99.193:45227)
0812 00:09:45.697962 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.697966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.697974 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:09:45.698025 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.698027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.698041 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.698489 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:45.698590 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.699239 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.700467 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.701229 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.701335 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.701393 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.702051 (+ 658us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.702097 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":82,"threads_started":1}
I20260812 00:09:45.702924 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.697355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:35007 (local address 127.0.0.1:50576)
0812 00:09:45.697704 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.697749 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.697819 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.698210 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.698211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.698212 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:45.699406 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.699408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.700706 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.700707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.701644 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.701646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.702696 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.702697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.702757 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.702760 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.702810 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.702848 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1131,"thread_start_us":77,"threads_started":1}
I20260812 00:09:45.702997 17830 tablet_service.cc:1467] Processing CreateTablet for tablet 7f14bec3c1bf410b80cb7018b783e7b2 (DEFAULT_TABLE table=test-table [id=ea65add3dada4d119ce04b20f7f8857d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:45.703143 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.697499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.16.99.194:35007)
0812 00:09:45.697910 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.697916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.697929 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:09:45.697975 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.697977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.697996 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.698088 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:45.699503 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.700582 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.701735 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.702541 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.702608 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.702660 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.702853 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.702901 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":121,"threads_started":1}
I20260812 00:09:45.703028 17539 tablet_service.cc:1467] Processing CreateTablet for tablet 7f14bec3c1bf410b80cb7018b783e7b2 (DEFAULT_TABLE table=test-table [id=ea65add3dada4d119ce04b20f7f8857d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:45.703426 17830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f14bec3c1bf410b80cb7018b783e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:45.703431 17539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f14bec3c1bf410b80cb7018b783e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:45.703752 17684 tablet_service.cc:1467] Processing CreateTablet for tablet 7f14bec3c1bf410b80cb7018b783e7b2 (DEFAULT_TABLE table=test-table [id=ea65add3dada4d119ce04b20f7f8857d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:45.704109 17684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f14bec3c1bf410b80cb7018b783e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:45.706254 17923 tablet_bootstrap.cc:492] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f: Bootstrap starting.
I20260812 00:09:45.707108 17924 tablet_bootstrap.cc:492] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b: Bootstrap starting.
I20260812 00:09:45.707755 17925 tablet_bootstrap.cc:492] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb: Bootstrap starting.
I20260812 00:09:45.707948 17923 tablet_bootstrap.cc:654] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:45.708021 17924 tablet_bootstrap.cc:654] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:45.708439 17925 tablet_bootstrap.cc:654] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:45.708830 17923 log.cc:824] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:45.708817 17924 log.cc:824] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:45.709092 17925 log.cc:824] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:45.709767 17923 tablet_bootstrap.cc:492] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f: No bootstrap required, opened a new log
I20260812 00:09:45.709848 17923 ts_tablet_manager.cc:1397] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 00:09:45.709998 17924 tablet_bootstrap.cc:492] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b: No bootstrap required, opened a new log
I20260812 00:09:45.710090 17924 ts_tablet_manager.cc:1397] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:09:45.710263 17925 tablet_bootstrap.cc:492] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb: No bootstrap required, opened a new log
I20260812 00:09:45.710402 17925 ts_tablet_manager.cc:1397] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:09:45.711995 17924 raft_consensus.cc:348] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.711997 17923 raft_consensus.cc:348] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.712200 17923 raft_consensus.cc:374] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:45.712201 17924 raft_consensus.cc:374] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:45.712249 17923 raft_consensus.cc:729] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04de1df424b4d6aa762e58014a0f51f, State: Initialized, Role: FOLLOWER
I20260812 00:09:45.712260 17924 raft_consensus.cc:729] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d008a9cde314c02b56a3629e45c843b, State: Initialized, Role: FOLLOWER
I20260812 00:09:45.712381 17923 consensus_queue.cc:260] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.712409 17924 consensus_queue.cc:260] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.712532 17925 raft_consensus.cc:348] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.712707 17925 raft_consensus.cc:374] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:45.712718 17923 ts_tablet_manager.cc:1428] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:09:45.712756 17925 raft_consensus.cc:729] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c241a47eb44f80a2e5d648d8f479cb, State: Initialized, Role: FOLLOWER
I20260812 00:09:45.712793 17924 ts_tablet_manager.cc:1428] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:09:45.712841 17906 heartbeater.cc:503] Master 127.16.99.254:33997 was elected leader, sending a full tablet report...
I20260812 00:09:45.713158 17925 consensus_queue.cc:260] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.712930 17612 heartbeater.cc:503] Master 127.16.99.254:33997 was elected leader, sending a full tablet report...
I20260812 00:09:45.713450 17925 ts_tablet_manager.cc:1428] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:09:45.713601 17759 heartbeater.cc:503] Master 127.16.99.254:33997 was elected leader, sending a full tablet report...
W20260812 00:09:45.734717 17760 tablet.cc:2369] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:09:45.813486 17613 tablet.cc:2369] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:09:45.913578 17907 tablet.cc:2369] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:09:45.971374 17931 raft_consensus.cc:482] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:45.971639 17931 raft_consensus.cc:504] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.972077 17931 leader_election.cc:288] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d008a9cde314c02b56a3629e45c843b (127.16.99.193:45227), e04de1df424b4d6aa762e58014a0f51f (127.16.99.195:36797)
I20260812 00:09:45.976240 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.972288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45227 (local address 127.16.99.194:33175)
0812 00:09:45.972388 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.972405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.972493 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.972743 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.972744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.972747 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:45.972840 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.972843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.973778 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.973779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.975126 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.975129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.975932 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.975933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.976030 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.976033 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.976123 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.976169 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":139}
I20260812 00:09:45.976255 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.972541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:48803 (local address 127.16.99.195:36797)
0812 00:09:45.972592 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.972594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.972711 (+ 117us) server_negotiation.cc:407] Connection header received
0812 00:09:45.972754 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.972755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.972767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.972849 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:45.973272 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.973862 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.975056 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.975556 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.975694 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.975705 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.975972 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.976034 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:09:45.976258 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.972417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:33175 (local address 127.16.99.193:45227)
0812 00:09:45.972526 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.972528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.972552 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:09:45.972596 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.972598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.972610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.972675 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:45.972957 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.973673 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.975333 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.975827 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.975917 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.975928 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.976140 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.976190 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 00:09:45.976518 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.972422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:36797 (local address 127.16.99.194:48803)
0812 00:09:45.972586 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:45.972597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:45.972665 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:45.972922 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:45.972924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:45.972927 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:45.973042 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.973044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.973965 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.973966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.974952 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:45.974955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.975754 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.975755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:45.975861 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.975867 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:09:45.976384 (+ 517us) client_negotiation.cc:240] Negotiation successful
0812 00:09:45.976437 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260812 00:09:45.976730 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f14bec3c1bf410b80cb7018b783e7b2" candidate_uuid: "40c241a47eb44f80a2e5d648d8f479cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d008a9cde314c02b56a3629e45c843b" is_pre_election: true
I20260812 00:09:45.976920 17562 raft_consensus.cc:2454] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c241a47eb44f80a2e5d648d8f479cb in term 0.
I20260812 00:09:45.977312 17633 leader_election.cc:302] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c241a47eb44f80a2e5d648d8f479cb, 7d008a9cde314c02b56a3629e45c843b; no voters:
I20260812 00:09:45.977401 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f14bec3c1bf410b80cb7018b783e7b2" candidate_uuid: "40c241a47eb44f80a2e5d648d8f479cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04de1df424b4d6aa762e58014a0f51f" is_pre_election: true
I20260812 00:09:45.977485 17931 raft_consensus.cc:2781] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:45.977553 17931 raft_consensus.cc:482] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:45.977591 17931 raft_consensus.cc:3037] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:45.977588 17853 raft_consensus.cc:2454] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c241a47eb44f80a2e5d648d8f479cb in term 0.
I20260812 00:09:45.978402 17931 raft_consensus.cc:504] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.978546 17931 leader_election.cc:288] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [CANDIDATE]: Term 1 election: Requested vote from peers 7d008a9cde314c02b56a3629e45c843b (127.16.99.193:45227), e04de1df424b4d6aa762e58014a0f51f (127.16.99.195:36797)
I20260812 00:09:45.978739 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f14bec3c1bf410b80cb7018b783e7b2" candidate_uuid: "40c241a47eb44f80a2e5d648d8f479cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04de1df424b4d6aa762e58014a0f51f"
I20260812 00:09:45.978744 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f14bec3c1bf410b80cb7018b783e7b2" candidate_uuid: "40c241a47eb44f80a2e5d648d8f479cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d008a9cde314c02b56a3629e45c843b"
I20260812 00:09:45.978818 17853 raft_consensus.cc:3037] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:45.978852 17562 raft_consensus.cc:3037] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:45.979635 17853 raft_consensus.cc:2454] T 7f14bec3c1bf410b80cb7018b783e7b2 P e04de1df424b4d6aa762e58014a0f51f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c241a47eb44f80a2e5d648d8f479cb in term 1.
I20260812 00:09:45.979671 17562 raft_consensus.cc:2454] T 7f14bec3c1bf410b80cb7018b783e7b2 P 7d008a9cde314c02b56a3629e45c843b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c241a47eb44f80a2e5d648d8f479cb in term 1.
I20260812 00:09:45.979862 17633 leader_election.cc:302] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c241a47eb44f80a2e5d648d8f479cb, e04de1df424b4d6aa762e58014a0f51f; no voters:
I20260812 00:09:45.979991 17931 raft_consensus.cc:2781] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:45.980197 17931 raft_consensus.cc:686] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [term 1 LEADER]: Becoming Leader. State: Replica: 40c241a47eb44f80a2e5d648d8f479cb, State: Running, Role: LEADER
I20260812 00:09:45.980304 17931 consensus_queue.cc:237] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } }
I20260812 00:09:45.981400 17401 catalog_manager.cc:5495] T 7f14bec3c1bf410b80cb7018b783e7b2 P 40c241a47eb44f80a2e5d648d8f479cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c241a47eb44f80a2e5d648d8f479cb (127.16.99.194). New cstate: current_term: 1 leader_uuid: "40c241a47eb44f80a2e5d648d8f479cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40c241a47eb44f80a2e5d648d8f479cb" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d008a9cde314c02b56a3629e45c843b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e04de1df424b4d6aa762e58014a0f51f" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36797 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:09:45 dist-test-slave-zpfg krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:09:45.996688 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:45.987619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.16.99.193:45227)
0812 00:09:45.987694 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:45.987696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:45.987839 (+ 143us) server_negotiation.cc:407] Connection header received
0812 00:09:45.987871 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:45.987872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:45.987883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:45.987961 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:45.988705 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.989228 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.990092 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:45.990432 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:45.990507 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:45.993283 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:45.993296 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:45.993296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:45.993305 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:45.995381 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.995820 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.995824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.995824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.995895 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:45.996100 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:45.996101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:45.996101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:45.996267 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:45.996387 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:45.996542 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 00:09:45.996616 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 00:09:46 dist-test-slave-zpfg krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:09:46 dist-test-slave-zpfg krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:09:46.024632 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:46.015655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.16.99.193:45227)
0812 00:09:46.015743 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:46.015745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:46.015879 (+ 134us) server_negotiation.cc:407] Connection header received
0812 00:09:46.015977 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:46.015978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:46.015996 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:46.016080 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:46.016772 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:46.017277 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:46.018093 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:46.018289 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:46.018408 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:46.021354 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:46.021355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:46.021356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:46.021372 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:46.023162 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:46.023690 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:46.023693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:46.023693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:46.023766 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:46.023988 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:46.023989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:46.023989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:46.024109 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:46.024223 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:46.024419 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 00:09:46.024513 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 00:09:46 dist-test-slave-zpfg krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:09:46 dist-test-slave-zpfg krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:09:46.052140 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:46.043550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.16.99.193:45227)
0812 00:09:46.043644 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:46.043647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:46.043861 (+ 214us) server_negotiation.cc:407] Connection header received
0812 00:09:46.043949 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:46.043950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:46.043962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:46.044053 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:46.044877 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:46.045538 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:46.046465 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:46.046657 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:46.046756 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:46.049449 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:46.049451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:46.049451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:46.049460 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:46.051045 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:46.051494 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:46.051495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:46.051495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:46.051549 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:46.051730 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:46.051731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:46.051731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:46.051826 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:46.051899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:46.051995 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 00:09:46.052055 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 00:09:46.052763 17517 tablet_service.cc:2908] Rejecting scan request for tablet 7f14bec3c1bf410b80cb7018b783e7b2: Uninitialized: clean time has not yet been initialized
I20260812 00:09:46.053900 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 17468
I20260812 00:09:46.061250 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 17617
I20260812 00:09:46.068213 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 17764
I20260812 00:09:46.073709 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 17366
2026-08-12T00:09:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2351 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:09:46 dist-test-slave-zpfg krb5kdc[17947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:09:46 dist-test-slave-zpfg krb5kdc[17947](info): set up 2 sockets
Aug 12 00:09:46 dist-test-slave-zpfg krb5kdc[17947](info): commencing operation
krb5kdc: starting...
W20260812 00:09:47.258966 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:09:47 dist-test-slave-zpfg krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:09:47.395723 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39391
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:47.483016 17961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:47.483253 17961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:47.483278 17961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:47.484440 17961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:09:47.484479 17961 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:09:47.484493 17961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:47.484506 17961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:47.484519 17961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:09:47.484530 17961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:09:47.486001 17961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:39391
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:47.486224 17961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:47.486492 17961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:47.491794 17969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:47.491871 17967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:47.491878 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:47.492038 17961 server_base.cc:1034] running on GCE node
I20260812 00:09:47.492450 17961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:47.492775 17961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:47.493954 17961 hybrid_clock.cc:648] HybridClock initialized: now 1786493387493940 us; error 48 us; skew 500 ppm
Aug 12 00:09:47 dist-test-slave-zpfg krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:47.496630 17961 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:09:47.497156 17961 webserver.cc:460] Webserver started at http://127.16.99.254:43427/ using document root <none> and password file <none>
I20260812 00:09:47.497429 17961 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:47.497501 17961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:47.497658 17961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:47.498931 17961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "782925ae0a544ffd823148ca7f1ff118"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "2f0325b4831816fa6959995f711e88af"
server_key_iv: "661beccd120fb1402f30191c7bc81c9a"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.499311 17961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "782925ae0a544ffd823148ca7f1ff118"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "2f0325b4831816fa6959995f711e88af"
server_key_iv: "661beccd120fb1402f30191c7bc81c9a"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.500731 17961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:09:47.501631 17977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:47.501817 17961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:47.501965 17961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "782925ae0a544ffd823148ca7f1ff118"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "2f0325b4831816fa6959995f711e88af"
server_key_iv: "661beccd120fb1402f30191c7bc81c9a"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.502060 17961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:47.535284 17961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:47.537878 17961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:47.538086 17961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:47.543324 17961 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:39391
I20260812 00:09:47.543407 18045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:39391 every 8 connection(s)
I20260812 00:09:47.543754 17961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:09:47.544587 18046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:47.547261 18046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118: Bootstrap starting.
I20260812 00:09:47.547947 18046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:47.548307 18046 log.cc:824] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:47.549341 18046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118: No bootstrap required, opened a new log
I20260812 00:09:47.551829 18046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782925ae0a544ffd823148ca7f1ff118" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39391 } }
I20260812 00:09:47.552026 18046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:47.552094 18046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782925ae0a544ffd823148ca7f1ff118, State: Initialized, Role: FOLLOWER
I20260812 00:09:47.552268 18046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782925ae0a544ffd823148ca7f1ff118" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39391 } }
I20260812 00:09:47.552361 18046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:09:47.552405 18046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:09:47.552464 18046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:47.552862 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 17961
I20260812 00:09:47.552943 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:09:47.553120 16783 external_mini_cluster.cc:1428] Setting key 05290f9ea9323cd04373b3755b34a285
I20260812 00:09:47.553237 18046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782925ae0a544ffd823148ca7f1ff118" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39391 } }
I20260812 00:09:47.553429 18046 leader_election.cc:302] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 782925ae0a544ffd823148ca7f1ff118; no voters:
I20260812 00:09:47.553645 18046 leader_election.cc:288] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:09:47.553697 18050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:47.553848 18050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [term 1 LEADER]: Becoming Leader. State: Replica: 782925ae0a544ffd823148ca7f1ff118, State: Running, Role: LEADER
I20260812 00:09:47.553941 18050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782925ae0a544ffd823148ca7f1ff118" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39391 } }
I20260812 00:09:47.553973 18046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:09:47.554313 18050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782925ae0a544ffd823148ca7f1ff118. Latest consensus state: current_term: 1 leader_uuid: "782925ae0a544ffd823148ca7f1ff118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782925ae0a544ffd823148ca7f1ff118" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39391 } } }
I20260812 00:09:47.554412 18050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:47.554454 18051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "782925ae0a544ffd823148ca7f1ff118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782925ae0a544ffd823148ca7f1ff118" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39391 } } }
I20260812 00:09:47.554572 18051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118 [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:47.554634 18057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:09:47.555183 18057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:09:47.558815 18057 catalog_manager.cc:1347] Generated new cluster ID: aebb90936064444ca2f7a4af9bc5a9d0
I20260812 00:09:47.558887 18057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:09:47 dist-test-slave-zpfg krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:47.565677 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.553813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.16.99.254:39391)
0812 00:09:47.554129 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:47.554139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:47.554174 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:09:47.554303 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:47.554310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:47.554547 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:47.555733 (+ 1186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:47.555843 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.557264 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.558199 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.558882 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.558995 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.561734 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:47.561739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:47.561739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:47.561754 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:47.563827 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.564272 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.564276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.564276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.564414 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.564678 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.564679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.564680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.564966 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:47.565189 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:47.565282 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 00:09:47.565450 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":75,"threads_started":1}
I20260812 00:09:47.592463 18057 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:09:47.593068 18057 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:09:47.601629 18057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118: Generated new TSK 0
I20260812 00:09:47.601872 18057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:09:47.636837 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39391
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:47.741721 18073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:47.741964 18073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:47.741991 18073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:47.743196 18073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:47.743259 18073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:47.743280 18073 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:09:47.743299 18073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:09:47.744729 18073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:47.744957 18073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:47.745146 18073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:47.750670 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:47.750766 18073 server_base.cc:1034] running on GCE node
W20260812 00:09:47.750659 18079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:47.750659 18078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:47.751171 18073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:47.751443 18073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:47.752625 18073 hybrid_clock.cc:648] HybridClock initialized: now 1786493387752628 us; error 47 us; skew 500 ppm
Aug 12 00:09:47 dist-test-slave-zpfg krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:47.755762 18073 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:09:47.756314 18073 webserver.cc:460] Webserver started at http://127.16.99.193:33289/ using document root <none> and password file <none>
I20260812 00:09:47.756567 18073 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:47.756671 18073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:47.756825 18073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:47.757937 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "cd2f8c0d3d5741099931655457cf07c8"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "dbd5770da764e93c381e820c804fafc5"
server_key_iv: "fdf4463f6940213138ea5e5093fb5cb9"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.758324 18073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "cd2f8c0d3d5741099931655457cf07c8"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "dbd5770da764e93c381e820c804fafc5"
server_key_iv: "fdf4463f6940213138ea5e5093fb5cb9"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.759828 18073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:47.760834 18089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:47.761117 18073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:47.761373 18073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "cd2f8c0d3d5741099931655457cf07c8"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "dbd5770da764e93c381e820c804fafc5"
server_key_iv: "fdf4463f6940213138ea5e5093fb5cb9"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.761476 18073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:47.775760 18073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:47.778752 18073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:47.778959 18073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:47.779613 18073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:47.779690 18073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:47.779743 18073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:47.779803 18073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:09:47 dist-test-slave-zpfg krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:47.788142 18073 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:36389
I20260812 00:09:47.788239 18210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:36389 every 8 connection(s)
I20260812 00:09:47.788662 18073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:09:47.791857 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.780622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:55891 (local address 127.16.99.254:39391)
0812 00:09:47.780718 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:47.780720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:47.780881 (+ 161us) server_negotiation.cc:407] Connection header received
0812 00:09:47.781572 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:47.781574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:47.781588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:47.781638 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:47.782721 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.783362 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.785278 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.785694 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.785787 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.788388 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:47.788390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:47.788391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:47.788401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:47.790238 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.790766 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.790767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.790767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.790875 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.791150 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.791151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.791151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.791277 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:47.791341 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:47.791680 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 00:09:47.791761 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:09:47.792228 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.780473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39391 (local address 127.16.99.193:55891)
0812 00:09:47.780723 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:47.780783 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:47.781432 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:47.781811 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:47.781816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:47.782280 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:47.782580 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.782587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.783468 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.783469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.785164 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.785168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.785800 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.785801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.785956 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.786599 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:47.786609 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:47.788266 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:47.790408 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.790416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.790418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.790679 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.790949 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.790950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.790950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.791077 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.791373 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:47.791381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:47.791530 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 00:09:47.791637 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 00:09:47.791794 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":114,"threads_started":1}
I20260812 00:09:47.792924 18211 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39391
I20260812 00:09:47.793015 18211 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:47.793207 18211 heartbeater.cc:511] Master 127.16.99.254:39391 requested a full tablet report, sending...
I20260812 00:09:47.793630 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 18073
I20260812 00:09:47.793733 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:09:47.793807 17995 ts_manager.cc:194] Registered new tserver with Master: cd2f8c0d3d5741099931655457cf07c8 (127.16.99.193:36389)
I20260812 00:09:47.793883 16783 external_mini_cluster.cc:1428] Setting key f1ff5d278d4ec3161234a826aa6585ef
I20260812 00:09:47.794864 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:55891
I20260812 00:09:47.800915 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.794203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:33217 (local address 127.16.99.254:39391)
0812 00:09:47.794244 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:47.794246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:47.794457 (+ 211us) server_negotiation.cc:407] Connection header received
0812 00:09:47.794561 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:47.794562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:47.794572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:47.794630 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:47.795293 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.795772 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.796516 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.796724 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.796798 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.797848 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:47.797849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:47.797849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:47.797857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:47.799634 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.800037 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.800038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.800038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.800101 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.800304 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.800305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.800306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.800433 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:47.800520 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:47.800728 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 00:09:47.800819 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:09:47.800925 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.794183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39391 (local address 127.16.99.193:33217)
0812 00:09:47.794388 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:47.794400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:47.794485 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:47.794636 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:47.794637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:47.794924 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:47.795006 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.795009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.795878 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.795879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.796418 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.796420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.796807 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.796808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.796892 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.797416 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:47.797421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:47.797735 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:47.799751 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.799753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.799753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.799946 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.800170 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.800171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.800171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.800235 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.800512 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:47.800513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:47.800574 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 00:09:47.800644 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:09:47.800802 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:09:47.825242 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39391
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:47.911904 18221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:47.912139 18221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:47.912168 18221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:47.913336 18221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:47.913376 18221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:47.913398 18221 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:09:47.913416 18221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:09:47.914892 18221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:47.915123 18221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:47.915316 18221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:47.920991 18227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:47.920995 18228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:47.921064 18221 server_base.cc:1034] running on GCE node
W20260812 00:09:47.920997 18230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:47.921655 18221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:47.921999 18221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:47.923183 18221 hybrid_clock.cc:648] HybridClock initialized: now 1786493387923163 us; error 46 us; skew 500 ppm
Aug 12 00:09:47 dist-test-slave-zpfg krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:47.926623 18221 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:09:47.927290 18221 webserver.cc:460] Webserver started at http://127.16.99.194:39469/ using document root <none> and password file <none>
I20260812 00:09:47.927560 18221 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:47.927636 18221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:47.927788 18221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:47.928932 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "ef6f8d3fff8cfc9d9995f0733413a697"
server_key_iv: "c8a3036c1432a0758767fa338fcf0cf6"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.929337 18221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "ef6f8d3fff8cfc9d9995f0733413a697"
server_key_iv: "c8a3036c1432a0758767fa338fcf0cf6"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.931084 18221 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 00:09:47.932049 18237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:47.932366 18221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:47.932480 18221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6"
format_stamp: "Formatted at 2026-08-12 00:09:47 on dist-test-slave-zpfg"
server_key: "ef6f8d3fff8cfc9d9995f0733413a697"
server_key_iv: "c8a3036c1432a0758767fa338fcf0cf6"
server_key_version: "encryptionkey@0"
I20260812 00:09:47.932547 18221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:47.951489 18221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:47.953737 18221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:47.953896 18221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:47.954509 18221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:47.954555 18221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:47.954578 18221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:47.954625 18221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:47.961865 18221 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:40383
I20260812 00:09:47.961947 18360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:40383 every 8 connection(s)
Aug 12 00:09:47 dist-test-slave-zpfg krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:47.962291 18221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:09:47.965870 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.954941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:45125 (local address 127.16.99.254:39391)
0812 00:09:47.955011 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:47.955013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:47.955257 (+ 244us) server_negotiation.cc:407] Connection header received
0812 00:09:47.955963 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:47.955965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:47.955979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:47.956052 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:47.957006 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.957862 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.959516 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.959856 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.959941 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.962903 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:47.962905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:47.962905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:47.962913 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:47.964468 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.964959 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.964960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.964960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.965014 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.965233 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.965234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.965234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.965390 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:47.965441 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:47.965710 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 00:09:47.965800 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:09:47.966192 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.954947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39391 (local address 127.16.99.194:45125)
0812 00:09:47.955122 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:47.955164 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:47.955838 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:47.956172 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:47.956175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:47.956578 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:47.956880 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.956888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.957989 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.957991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.959409 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.959414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.959944 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.959945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.960051 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.960619 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:47.960630 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:47.962790 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:47.964558 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.964565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.964567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.964883 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.965073 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.965074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.965074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.965191 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.965452 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:47.965458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:47.965594 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 00:09:47.965683 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 00:09:47.965854 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260812 00:09:47.966830 18362 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39391
I20260812 00:09:47.966919 18362 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:47.967125 18362 heartbeater.cc:511] Master 127.16.99.254:39391 requested a full tablet report, sending...
I20260812 00:09:47.967583 17995 ts_manager.cc:194] Registered new tserver with Master: be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383)
I20260812 00:09:47.968125 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:45125
I20260812 00:09:47.971827 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 18221
I20260812 00:09:47.971932 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:09:47.972136 16783 external_mini_cluster.cc:1428] Setting key c545a715d5a6d6b7b3bfda591e398cbd
I20260812 00:09:47.974325 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.968042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:57251 (local address 127.16.99.254:39391)
0812 00:09:47.968092 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:47.968094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:47.968178 (+ 84us) server_negotiation.cc:407] Connection header received
0812 00:09:47.968242 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:47.968242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:47.968253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:47.968313 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:47.968726 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.969215 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.969949 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.970158 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.970243 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.971198 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:47.971200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:47.971201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:47.971208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:47.972937 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.973398 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.973399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.973400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.973464 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:47.973687 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:47.973688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:47.973688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:47.973866 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:47.973925 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:47.974097 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 00:09:47.974168 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:09:47.974398 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:47.967951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39391 (local address 127.16.99.194:57251)
0812 00:09:47.968111 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:47.968123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:47.968189 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:47.968331 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:47.968331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:47.968557 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:47.968644 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.968646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.969329 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.969331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.969863 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:47.969864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:47.970246 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:47.970247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:47.970362 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:47.970814 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:47.970818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:47.971105 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:47.973018 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.973019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.973020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.973263 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.973543 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:47.973544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:47.973545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:47.973621 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:47.973937 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:47.973938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:47.973998 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 00:09:47.974073 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 00:09:47.974208 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:09:48.002890 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39391
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:48.091638 18372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:48.091898 18372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:48.091945 18372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:48.093300 18372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:48.093375 18372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:48.093412 18372 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:09:48.093442 18372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:09:48.095077 18372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:48.095376 18372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:48.095624 18372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:48.101152 18381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:48.101292 18379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:48.101169 18378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:48.101594 18372 server_base.cc:1034] running on GCE node
I20260812 00:09:48.101814 18372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:48.102092 18372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:48.103260 18372 hybrid_clock.cc:648] HybridClock initialized: now 1786493388103240 us; error 38 us; skew 500 ppm
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493388, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:48.105981 18372 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:09:48.106594 18372 webserver.cc:460] Webserver started at http://127.16.99.195:45879/ using document root <none> and password file <none>
I20260812 00:09:48.106853 18372 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:48.106927 18372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:48.107087 18372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:48.108194 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "587367586b4d4f3586fe10e73428a5c6"
format_stamp: "Formatted at 2026-08-12 00:09:48 on dist-test-slave-zpfg"
server_key: "3cd78cf161ef9aca225dd61c42127cdb"
server_key_iv: "97d66c697e8ef507c8e88a38991923cb"
server_key_version: "encryptionkey@0"
I20260812 00:09:48.108634 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "587367586b4d4f3586fe10e73428a5c6"
format_stamp: "Formatted at 2026-08-12 00:09:48 on dist-test-slave-zpfg"
server_key: "3cd78cf161ef9aca225dd61c42127cdb"
server_key_iv: "97d66c697e8ef507c8e88a38991923cb"
server_key_version: "encryptionkey@0"
I20260812 00:09:48.110095 18372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:09:48.111033 18389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:48.111238 18372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:09:48.111379 18372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "587367586b4d4f3586fe10e73428a5c6"
format_stamp: "Formatted at 2026-08-12 00:09:48 on dist-test-slave-zpfg"
server_key: "3cd78cf161ef9aca225dd61c42127cdb"
server_key_iv: "97d66c697e8ef507c8e88a38991923cb"
server_key_version: "encryptionkey@0"
I20260812 00:09:48.111467 18372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:48.131376 18372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:48.134047 18372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:48.134239 18372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:48.134958 18372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:48.135020 18372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:48.135058 18372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:48.135083 18372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493388, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:48.143131 18372 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:32901
I20260812 00:09:48.143288 18514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:32901 every 8 connection(s)
I20260812 00:09:48.143611 18372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:09:48.146579 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.135451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:35345 (local address 127.16.99.254:39391)
0812 00:09:48.135522 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.135525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.135882 (+ 357us) server_negotiation.cc:407] Connection header received
0812 00:09:48.136706 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.136708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.136723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.136795 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:48.137811 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.138539 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.140259 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.140486 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.140550 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.143638 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:48.143640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:48.143640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:48.143649 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:48.145233 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.145653 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.145654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.145654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.145710 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.145936 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.145936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.145937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.146047 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:48.146102 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.146427 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.146513 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:09:48.146760 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.135457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39391 (local address 127.16.99.195:35345)
0812 00:09:48.135696 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.135763 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.136557 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.136881 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.136886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.137305 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:48.137656 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.137663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.138640 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.138641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.140147 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.140150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.140574 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.140575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.140768 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.141500 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:48.141512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:48.143518 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:48.145332 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:48.145339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:48.145340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:48.145576 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:48.145772 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:48.145773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:48.145773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:48.145880 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:48.146112 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:48.146118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:48.146248 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.146352 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.146484 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":128,"threads_started":1}
I20260812 00:09:48.147310 18515 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39391
I20260812 00:09:48.147399 18515 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:48.147595 18515 heartbeater.cc:511] Master 127.16.99.254:39391 requested a full tablet report, sending...
I20260812 00:09:48.147997 17995 ts_manager.cc:194] Registered new tserver with Master: 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195:32901)
I20260812 00:09:48.148516 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:35345
I20260812 00:09:48.149641 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 18372
I20260812 00:09:48.149712 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:09:48.149885 16783 external_mini_cluster.cc:1428] Setting key 16fda6db4bc5b0e00877fc36683856f1
I20260812 00:09:48.150898 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:09:48.155519 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.148569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39391 (local address 127.16.99.195:44973)
0812 00:09:48.148766 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.148777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.148836 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.148956 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.148957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.149235 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:48.149303 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.149305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.150146 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.150147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.150969 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.150972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.151309 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.151310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.151400 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.151902 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:48.151908 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:48.152354 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:48.154246 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:48.154247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:48.154248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:48.154496 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:48.154711 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:48.154712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:48.154712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:48.154760 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:48.155113 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:48.155114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:48.155168 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.155248 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.155377 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260812 00:09:48.155905 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.148657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:44973 (local address 127.16.99.254:39391)
0812 00:09:48.148701 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.148703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.148850 (+ 147us) server_negotiation.cc:407] Connection header received
0812 00:09:48.148878 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.148879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.148892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.148952 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:48.149399 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.150042 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.151036 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.151234 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.151289 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.152452 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:48.152453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:48.152453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:48.152460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:48.154163 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.154576 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.154577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.154577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.154642 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.154832 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.154833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.154833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.155007 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:48.155082 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.155670 (+ 588us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.155766 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[17947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:48.179698 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.171620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.16.99.254:39391)
0812 00:09:48.171694 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.171696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.171728 (+ 32us) server_negotiation.cc:407] Connection header received
0812 00:09:48.171764 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.171765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.171775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.171832 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:48.172521 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.173061 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.173847 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.174050 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.174141 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.176942 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:48.176944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:48.176944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:48.176954 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:48.178435 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.178810 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.178812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.178812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.178876 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.179085 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.179086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.179086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.179233 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:48.179359 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.179491 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.179582 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:09:48.181931 17995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:09:48.183768 17995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:48.191483 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.186823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:36389 (local address 127.0.0.1:37758)
0812 00:09:48.187391 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.187403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.187447 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.187713 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.187714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.187716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:48.187749 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.187750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.188791 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.188792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.190105 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.190108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.191112 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.191113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.191214 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.191231 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.191322 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.191385 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":179,"thread_start_us":118,"threads_started":1}
I20260812 00:09:48.191526 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.187111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.16.99.193:36389)
0812 00:09:48.187477 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.187483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.187495 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:09:48.187539 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.187541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.187558 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.187641 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:48.187839 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.188682 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.190201 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.191011 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.191101 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.191166 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.191351 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.191399 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":83,"threads_started":1}
I20260812 00:09:48.191874 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.186788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:40383 (local address 127.0.0.1:51708)
0812 00:09:48.187232 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.187245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.187301 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.187627 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.187631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.187633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:48.187731 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.187734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.189131 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.189132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.190550 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.190554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.191592 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.191593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.191679 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.191684 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.191738 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.191787 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":60,"thread_start_us":119,"threads_started":1}
I20260812 00:09:48.192227 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.187021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.16.99.194:40383)
0812 00:09:48.187388 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.187393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.187405 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:09:48.187450 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.187452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.187468 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.187553 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:48.187825 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.189003 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.190649 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.191409 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.191481 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.191539 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.192049 (+ 510us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.192100 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":108,"threads_started":1}
I20260812 00:09:48.192297 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.186788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:32901 (local address 127.0.0.1:34956)
0812 00:09:48.187124 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.187179 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.187259 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.187862 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.187863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.187865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:48.187900 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.187902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.189147 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.189148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.190642 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.190645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.191905 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.191906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.192006 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.192012 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.192060 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.192101 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":96,"thread_start_us":79,"threads_started":1}
I20260812 00:09:48.192699 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.187021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.16.99.195:32901)
0812 00:09:48.187706 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.187710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.187718 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:09:48.187749 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.187751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.187763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.187867 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:48.187967 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.189051 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.190745 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.191491 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.191569 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.192492 (+ 923us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.192524 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.192575 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":214,"threads_started":1}
I20260812 00:09:48.192631 18141 tablet_service.cc:1467] Processing CreateTablet for tablet c621cdb9b73b4695980781d38c2fdedf (DEFAULT_TABLE table=test-table [id=06298965117f4b1cb51b0026c4ada48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:48.192961 18289 tablet_service.cc:1467] Processing CreateTablet for tablet c621cdb9b73b4695980781d38c2fdedf (DEFAULT_TABLE table=test-table [id=06298965117f4b1cb51b0026c4ada48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:48.193115 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c621cdb9b73b4695980781d38c2fdedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.193320 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c621cdb9b73b4695980781d38c2fdedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.193377 18434 tablet_service.cc:1467] Processing CreateTablet for tablet c621cdb9b73b4695980781d38c2fdedf (DEFAULT_TABLE table=test-table [id=06298965117f4b1cb51b0026c4ada48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:48.193718 18434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c621cdb9b73b4695980781d38c2fdedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.196463 18539 tablet_bootstrap.cc:492] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Bootstrap starting.
I20260812 00:09:48.197263 18539 tablet_bootstrap.cc:654] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.197443 18540 tablet_bootstrap.cc:492] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Bootstrap starting.
I20260812 00:09:48.197683 18539 log.cc:824] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:48.198041 18541 tablet_bootstrap.cc:492] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Bootstrap starting.
I20260812 00:09:48.198184 18540 tablet_bootstrap.cc:654] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.198621 18540 log.cc:824] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:48.198683 18539 tablet_bootstrap.cc:492] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: No bootstrap required, opened a new log
I20260812 00:09:48.198760 18539 ts_tablet_manager.cc:1397] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:09:48.199043 18541 tablet_bootstrap.cc:654] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.199412 18541 log.cc:824] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:48.199561 18540 tablet_bootstrap.cc:492] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: No bootstrap required, opened a new log
I20260812 00:09:48.199689 18540 ts_tablet_manager.cc:1397] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:09:48.200268 18541 tablet_bootstrap.cc:492] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: No bootstrap required, opened a new log
I20260812 00:09:48.200387 18541 ts_tablet_manager.cc:1397] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:09:48.200883 18539 raft_consensus.cc:348] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.201061 18539 raft_consensus.cc:374] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.201112 18539 raft_consensus.cc:729] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2f8c0d3d5741099931655457cf07c8, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.201231 18539 consensus_queue.cc:260] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.201586 18211 heartbeater.cc:503] Master 127.16.99.254:39391 was elected leader, sending a full tablet report...
I20260812 00:09:48.201946 18539 ts_tablet_manager.cc:1428] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:09:48.201875 18540 raft_consensus.cc:348] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.202044 18540 raft_consensus.cc:374] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.202080 18540 raft_consensus.cc:729] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be48d365ffcc4c8d90e9c3c2b2d831f6, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.202195 18540 consensus_queue.cc:260] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.202499 18540 ts_tablet_manager.cc:1428] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:09:48.202576 18362 heartbeater.cc:503] Master 127.16.99.254:39391 was elected leader, sending a full tablet report...
I20260812 00:09:48.202610 18541 raft_consensus.cc:348] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.202739 18541 raft_consensus.cc:374] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.202759 18541 raft_consensus.cc:729] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587367586b4d4f3586fe10e73428a5c6, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.202872 18541 consensus_queue.cc:260] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.203168 18541 ts_tablet_manager.cc:1428] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:09:48.203303 18515 heartbeater.cc:503] Master 127.16.99.254:39391 was elected leader, sending a full tablet report...
W20260812 00:09:48.212952 18363 tablet.cc:2369] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:09:48.220270 18545 raft_consensus.cc:482] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:48.220418 18545 raft_consensus.cc:504] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.220733 18545 leader_election.cc:288] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383), 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195:32901)
I20260812 00:09:48.223898 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.220970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:40383 (local address 127.16.99.193:42749)
0812 00:09:48.221039 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.221056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.221115 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.221398 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.221399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.221401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:48.221479 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.221480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.222140 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.222141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.223014 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.223016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.223606 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.223607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.223706 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.223710 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.223775 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.223818 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 00:09:48.223897 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.221069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:42749 (local address 127.16.99.194:40383)
0812 00:09:48.221122 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.221124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.221276 (+ 152us) server_negotiation.cc:407] Connection header received
0812 00:09:48.221303 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.221304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.221312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.221390 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:48.221555 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.222053 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.223109 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.223502 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.223604 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.223613 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.223793 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.223831 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 00:09:48.224284 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c621cdb9b73b4695980781d38c2fdedf" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" is_pre_election: true
I20260812 00:09:48.224385 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.221155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:36843 (local address 127.16.99.195:32901)
0812 00:09:48.221201 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.221204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.221286 (+ 82us) server_negotiation.cc:407] Connection header received
0812 00:09:48.221517 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.221518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.221531 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.221617 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:48.221821 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.222461 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.223508 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.223943 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.224046 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.224058 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.224274 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.224320 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:09:48.224395 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.220971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:32901 (local address 127.16.99.193:36843)
0812 00:09:48.221212 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.221221 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.221277 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.221636 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.221637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.221640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:48.221738 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.221740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.222560 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.222562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.223402 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.223404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.224089 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.224090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.224190 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.224193 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.224265 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.224301 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":24,"thread_start_us":139,"threads_started":1}
I20260812 00:09:48.224504 18309 raft_consensus.cc:2454] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 0.
I20260812 00:09:48.224704 18463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c621cdb9b73b4695980781d38c2fdedf" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587367586b4d4f3586fe10e73428a5c6" is_pre_election: true
I20260812 00:09:48.224797 18092 leader_election.cc:302] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be48d365ffcc4c8d90e9c3c2b2d831f6, cd2f8c0d3d5741099931655457cf07c8; no voters:
I20260812 00:09:48.224865 18463 raft_consensus.cc:2454] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 0.
I20260812 00:09:48.224943 18545 raft_consensus.cc:2781] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:48.225035 18545 raft_consensus.cc:482] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:48.225076 18545 raft_consensus.cc:3037] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.226256 18545 raft_consensus.cc:504] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.226466 18545 leader_election.cc:288] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 election: Requested vote from peers be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383), 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195:32901)
I20260812 00:09:48.226646 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c621cdb9b73b4695980781d38c2fdedf" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6"
I20260812 00:09:48.226763 18463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c621cdb9b73b4695980781d38c2fdedf" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587367586b4d4f3586fe10e73428a5c6"
I20260812 00:09:48.226809 18309 raft_consensus.cc:3037] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.226846 18463 raft_consensus.cc:3037] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.227833 18463 raft_consensus.cc:2454] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 1.
I20260812 00:09:48.227833 18309 raft_consensus.cc:2454] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 1.
I20260812 00:09:48.228061 18092 leader_election.cc:302] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be48d365ffcc4c8d90e9c3c2b2d831f6, cd2f8c0d3d5741099931655457cf07c8; no voters:
I20260812 00:09:48.228319 18545 raft_consensus.cc:2781] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:48.228498 18545 raft_consensus.cc:686] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 1 LEADER]: Becoming Leader. State: Replica: cd2f8c0d3d5741099931655457cf07c8, State: Running, Role: LEADER
I20260812 00:09:48.228609 18545 consensus_queue.cc:237] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.229694 17996 catalog_manager.cc:5495] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd2f8c0d3d5741099931655457cf07c8 (127.16.99.193). New cstate: current_term: 1 leader_uuid: "cd2f8c0d3d5741099931655457cf07c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } health_report { overall_health: UNKNOWN } } }
I20260812 00:09:48.241757 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.239028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.16.99.193:36389)
0812 00:09:48.239093 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.239096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.239162 (+ 66us) server_negotiation.cc:407] Connection header received
0812 00:09:48.239345 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.239347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.239367 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.239439 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:48.239636 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.240120 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.240961 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.241156 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.241226 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.241428 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:48.241524 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:48.241585 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.241673 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.241693 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:09:48.244041 18309 raft_consensus.cc:1261] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Refusing update from remote peer cd2f8c0d3d5741099931655457cf07c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:48.244041 18463 raft_consensus.cc:1261] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Refusing update from remote peer cd2f8c0d3d5741099931655457cf07c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:48.244474 18549 consensus_queue.cc:1035] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:48.244664 18545 consensus_queue.cc:1035] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:48.247520 18553 mvcc.cc:205] Tried to move back new op lower bound from 7317476918244098048 to 7317476918185480192. Current Snapshot: MvccSnapshot[applied={T|T < 7317476918244098048}]
I20260812 00:09:48.247851 18554 mvcc.cc:205] Tried to move back new op lower bound from 7317476918244098048 to 7317476918185480192. Current Snapshot: MvccSnapshot[applied={T|T < 7317476918244098048}]
I20260812 00:09:48.251765 17996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:48.251896 17996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:48.252962 17996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118: Sending DeleteTablet for 3 replicas of tablet c621cdb9b73b4695980781d38c2fdedf
I20260812 00:09:48.253319 18141 tablet_service.cc:1514] Processing DeleteTablet for tablet c621cdb9b73b4695980781d38c2fdedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:48 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:37758
I20260812 00:09:48.253360 18434 tablet_service.cc:1514] Processing DeleteTablet for tablet c621cdb9b73b4695980781d38c2fdedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:48 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34956
I20260812 00:09:48.253597 18289 tablet_service.cc:1514] Processing DeleteTablet for tablet c621cdb9b73b4695980781d38c2fdedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:48 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51708
I20260812 00:09:48.254074 18563 tablet_replica.cc:330] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: stopping tablet replica
I20260812 00:09:48.254163 18564 tablet_replica.cc:330] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: stopping tablet replica
I20260812 00:09:48.254204 17996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 00:09:48.254384 18564 raft_consensus.cc:2229] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 00:09:48.254426 17996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:48.254527 18564 raft_consensus.cc:2258] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:48.255199 18564 ts_tablet_manager.cc:1905] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:48.255908 18141 tablet_service.cc:1467] Processing CreateTablet for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 (DEFAULT_TABLE table=test-table [id=b153a27a03bc456fb96ae82430e9ff84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:48.256114 18289 tablet_service.cc:1467] Processing CreateTablet for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 (DEFAULT_TABLE table=test-table [id=b153a27a03bc456fb96ae82430e9ff84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:48.256172 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.256317 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.257201 18564 ts_tablet_manager.cc:1918] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:09:48.257272 18564 log.cc:1196] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/c621cdb9b73b4695980781d38c2fdedf
I20260812 00:09:48.257620 18539 tablet_bootstrap.cc:492] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Bootstrap starting.
I20260812 00:09:48.257637 18540 tablet_bootstrap.cc:492] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: Bootstrap starting.
I20260812 00:09:48.254977 18563 raft_consensus.cc:2229] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:09:48.258087 18540 tablet_bootstrap.cc:654] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.258090 18539 tablet_bootstrap.cc:654] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.258248 18563 raft_consensus.cc:2258] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:48.258453 18562 tablet_replica.cc:330] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: stopping tablet replica
I20260812 00:09:48.258584 18562 raft_consensus.cc:2229] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:48.258735 18562 raft_consensus.cc:2258] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:48.258870 18563 ts_tablet_manager.cc:1905] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:48.259037 18434 tablet_service.cc:1467] Processing CreateTablet for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 (DEFAULT_TABLE table=test-table [id=b153a27a03bc456fb96ae82430e9ff84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:48.259163 18434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.259225 18562 ts_tablet_manager.cc:1905] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:48.260452 18563 ts_tablet_manager.cc:1918] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:09:48.260520 18563 log.cc:1196] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/c621cdb9b73b4695980781d38c2fdedf
I20260812 00:09:48.260674 18541 tablet_bootstrap.cc:492] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: Bootstrap starting.
I20260812 00:09:48.260919 18563 ts_tablet_manager.cc:1939] T c621cdb9b73b4695980781d38c2fdedf P cd2f8c0d3d5741099931655457cf07c8: Deleting consensus metadata
I20260812 00:09:48.257551 18564 ts_tablet_manager.cc:1939] T c621cdb9b73b4695980781d38c2fdedf P be48d365ffcc4c8d90e9c3c2b2d831f6: Deleting consensus metadata
I20260812 00:09:48.261093 18541 tablet_bootstrap.cc:654] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.261263 18562 ts_tablet_manager.cc:1918] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:09:48.261327 18562 log.cc:1196] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/c621cdb9b73b4695980781d38c2fdedf
I20260812 00:09:48.261538 17979 catalog_manager.cc:4841] TS be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383): tablet c621cdb9b73b4695980781d38c2fdedf (table test-table [id=06298965117f4b1cb51b0026c4ada48b]) successfully deleted
I20260812 00:09:48.261581 18562 ts_tablet_manager.cc:1939] T c621cdb9b73b4695980781d38c2fdedf P 587367586b4d4f3586fe10e73428a5c6: Deleting consensus metadata
I20260812 00:09:48.261969 18541 tablet_bootstrap.cc:492] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: No bootstrap required, opened a new log
I20260812 00:09:48.262037 18541 ts_tablet_manager.cc:1397] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:48.262192 18541 raft_consensus.cc:348] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.262274 18541 raft_consensus.cc:374] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.262265 17981 catalog_manager.cc:4841] TS 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195:32901): tablet c621cdb9b73b4695980781d38c2fdedf (table test-table [id=06298965117f4b1cb51b0026c4ada48b]) successfully deleted
I20260812 00:09:48.262301 18541 raft_consensus.cc:729] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587367586b4d4f3586fe10e73428a5c6, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.262372 18541 consensus_queue.cc:260] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.262466 18541 ts_tablet_manager.cc:1428] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:48.262557 17978 catalog_manager.cc:4841] TS cd2f8c0d3d5741099931655457cf07c8 (127.16.99.193:36389): tablet c621cdb9b73b4695980781d38c2fdedf (table test-table [id=06298965117f4b1cb51b0026c4ada48b]) successfully deleted
I20260812 00:09:48.262653 18540 tablet_bootstrap.cc:492] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: No bootstrap required, opened a new log
I20260812 00:09:48.262706 18540 ts_tablet_manager.cc:1397] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 00:09:48.262916 18540 raft_consensus.cc:348] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.263029 18540 raft_consensus.cc:374] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.263055 18540 raft_consensus.cc:729] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be48d365ffcc4c8d90e9c3c2b2d831f6, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.263098 18540 consensus_queue.cc:260] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.263191 18540 ts_tablet_manager.cc:1428] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:09:48.263641 18539 tablet_bootstrap.cc:492] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: No bootstrap required, opened a new log
I20260812 00:09:48.263701 18539 ts_tablet_manager.cc:1397] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260812 00:09:48.263873 18539 raft_consensus.cc:348] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.263953 18539 raft_consensus.cc:374] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.263973 18539 raft_consensus.cc:729] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2f8c0d3d5741099931655457cf07c8, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.264038 18539 consensus_queue.cc:260] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.264117 18539 ts_tablet_manager.cc:1428] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260812 00:09:48.289613 18212 tablet.cc:2369] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:09:48.312482 18557 raft_consensus.cc:482] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:48.312604 18557 raft_consensus.cc:504] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.312784 18557 leader_election.cc:288] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383), 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195:32901)
I20260812 00:09:48.313052 18463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f69631ac4f4f1f856bc2e0f5c9dde4" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587367586b4d4f3586fe10e73428a5c6" is_pre_election: true
I20260812 00:09:48.313171 18463 raft_consensus.cc:2454] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 0.
I20260812 00:09:48.313050 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f69631ac4f4f1f856bc2e0f5c9dde4" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" is_pre_election: true
I20260812 00:09:48.313284 18309 raft_consensus.cc:2454] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 0.
I20260812 00:09:48.313377 18093 leader_election.cc:302] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 587367586b4d4f3586fe10e73428a5c6, cd2f8c0d3d5741099931655457cf07c8; no voters:
I20260812 00:09:48.313469 18557 raft_consensus.cc:2781] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:48.313495 18557 raft_consensus.cc:482] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:48.313515 18557 raft_consensus.cc:3037] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.314442 18557 raft_consensus.cc:504] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.314611 18557 leader_election.cc:288] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 election: Requested vote from peers be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383), 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195:32901)
I20260812 00:09:48.314819 18463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f69631ac4f4f1f856bc2e0f5c9dde4" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587367586b4d4f3586fe10e73428a5c6"
I20260812 00:09:48.314893 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f69631ac4f4f1f856bc2e0f5c9dde4" candidate_uuid: "cd2f8c0d3d5741099931655457cf07c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6"
I20260812 00:09:48.314944 18463 raft_consensus.cc:3037] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.314983 18309 raft_consensus.cc:3037] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.315894 18463 raft_consensus.cc:2454] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 1.
I20260812 00:09:48.315905 18309 raft_consensus.cc:2454] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd2f8c0d3d5741099931655457cf07c8 in term 1.
I20260812 00:09:48.316129 18092 leader_election.cc:302] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be48d365ffcc4c8d90e9c3c2b2d831f6, cd2f8c0d3d5741099931655457cf07c8; no voters:
I20260812 00:09:48.316222 18557 raft_consensus.cc:2781] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:48.316267 18557 raft_consensus.cc:686] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 1 LEADER]: Becoming Leader. State: Replica: cd2f8c0d3d5741099931655457cf07c8, State: Running, Role: LEADER
I20260812 00:09:48.316321 18557 consensus_queue.cc:237] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.316996 17996 catalog_manager.cc:5495] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd2f8c0d3d5741099931655457cf07c8 (127.16.99.193). New cstate: current_term: 1 leader_uuid: "cd2f8c0d3d5741099931655457cf07c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:48.375547 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.367413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.16.99.254:39391)
0812 00:09:48.367479 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.367481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.367563 (+ 82us) server_negotiation.cc:407] Connection header received
0812 00:09:48.367690 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.367691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.367701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.367774 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:48.368265 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.368763 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.369473 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.369668 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.369768 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.372663 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:48.372665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:48.372666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:48.372675 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:48.374228 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.374689 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.374690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.374691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.374750 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.374943 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.374944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.374944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.375066 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:48.375142 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.375302 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.375395 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:09:48.375609 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.367317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39391 (local address 127.0.0.1:52072)
0812 00:09:48.367484 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.367499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.367582 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.367779 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.367780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.368119 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:48.368173 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.368176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.368852 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.368854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.369386 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.369388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.369772 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.369773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.369836 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.370646 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:48.370652 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:48.372548 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:48.374381 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:48.374392 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:48.374394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:48.374608 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:48.374817 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:48.374817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:48.374818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:48.374880 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:48.375140 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:48.375149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:48.375195 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.375278 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.375476 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":79,"thread_start_us":90,"threads_started":1}
I20260812 00:09:48.379695 17996 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:52072:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:09:48.380081 17996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:48.382125 18141 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6be27108e94908b16d946582bb7284 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9655b598960546eb92f060f994f2cca6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:09:48.382181 18434 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6be27108e94908b16d946582bb7284 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9655b598960546eb92f060f994f2cca6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:09:48.382332 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6be27108e94908b16d946582bb7284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.382368 18434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6be27108e94908b16d946582bb7284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.382319 18289 tablet_service.cc:1467] Processing CreateTablet for tablet 5b6be27108e94908b16d946582bb7284 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9655b598960546eb92f060f994f2cca6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:09:48.382525 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b6be27108e94908b16d946582bb7284. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:48.384073 18540 tablet_bootstrap.cc:492] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: Bootstrap starting.
I20260812 00:09:48.384539 18540 tablet_bootstrap.cc:654] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.384867 18539 tablet_bootstrap.cc:492] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: Bootstrap starting.
I20260812 00:09:48.385216 18539 tablet_bootstrap.cc:654] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.385279 18540 tablet_bootstrap.cc:492] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: No bootstrap required, opened a new log
I20260812 00:09:48.385407 18540 ts_tablet_manager.cc:1397] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:48.385507 18541 tablet_bootstrap.cc:492] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: Bootstrap starting.
I20260812 00:09:48.385615 18540 raft_consensus.cc:348] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.385712 18540 raft_consensus.cc:374] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.385740 18540 raft_consensus.cc:729] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be48d365ffcc4c8d90e9c3c2b2d831f6, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.385799 18540 consensus_queue.cc:260] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.385900 18540 ts_tablet_manager.cc:1428] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:48.385993 18539 tablet_bootstrap.cc:492] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: No bootstrap required, opened a new log
I20260812 00:09:48.385980 18546 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.386024 18541 tablet_bootstrap.cc:654] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:48.386044 18539 ts_tablet_manager.cc:1397] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:48.386060 18546 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:48.386197 18539 raft_consensus.cc:348] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.386286 18539 raft_consensus.cc:374] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.386307 18539 raft_consensus.cc:729] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2f8c0d3d5741099931655457cf07c8, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.386382 18539 consensus_queue.cc:260] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.386502 18539 ts_tablet_manager.cc:1428] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:48.386533 18557 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.386649 18557 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:48.386875 18541 tablet_bootstrap.cc:492] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: No bootstrap required, opened a new log
I20260812 00:09:48.386919 18541 ts_tablet_manager.cc:1397] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:48.387061 18541 raft_consensus.cc:348] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.387133 18541 raft_consensus.cc:374] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:48.387154 18541 raft_consensus.cc:729] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587367586b4d4f3586fe10e73428a5c6, State: Initialized, Role: FOLLOWER
I20260812 00:09:48.387208 18541 consensus_queue.cc:260] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.387307 18541 ts_tablet_manager.cc:1428] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:48.387414 18547 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.387540 18547 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 00:09:48.394263 18516 tablet.cc:2369] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:09:48.469908 18547 raft_consensus.cc:482] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:48.470036 18547 raft_consensus.cc:504] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.470487 18547 leader_election.cc:288] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd2f8c0d3d5741099931655457cf07c8 (127.16.99.193:36389), be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383)
I20260812 00:09:48.474036 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.470692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:40383 (local address 127.16.99.195:42289)
0812 00:09:48.470776 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.470795 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.470858 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.471183 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.471185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.471187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:48.471285 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.471287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.472091 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.472092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.472896 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.472898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.473701 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.473702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.473832 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.473837 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.473903 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.473950 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 00:09:48.474074 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.470937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:42289 (local address 127.16.99.194:40383)
0812 00:09:48.471039 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.471041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.471052 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:09:48.471088 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.471089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.471101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.471167 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:48.471388 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.471964 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.473002 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.473614 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.473685 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.473693 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.473957 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.474014 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:09:48.474409 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6be27108e94908b16d946582bb7284" candidate_uuid: "587367586b4d4f3586fe10e73428a5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" is_pre_election: true
I20260812 00:09:48.474531 18309 raft_consensus.cc:2454] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587367586b4d4f3586fe10e73428a5c6 in term 0.
I20260812 00:09:48.474550 18579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.470766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:36389 (local address 127.16.99.195:55501)
0812 00:09:48.470976 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.470988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.471072 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.471277 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.471278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.471280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:48.471398 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.471400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.472036 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.472038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.473407 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.473410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.474212 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.474216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.474296 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.474300 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.474403 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.474453 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":142,"threads_started":1}
I20260812 00:09:48.474588 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.470807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:55501 (local address 127.16.99.193:36389)
0812 00:09:48.470899 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.470901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.471083 (+ 182us) server_negotiation.cc:407] Connection header received
0812 00:09:48.471164 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.471165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.471181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.471243 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:48.471487 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.471946 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.473516 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.474060 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.474134 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.474144 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.474434 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.474492 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 00:09:48.474789 18391 leader_election.cc:302] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 587367586b4d4f3586fe10e73428a5c6, be48d365ffcc4c8d90e9c3c2b2d831f6; no voters:
I20260812 00:09:48.474967 18547 raft_consensus.cc:2781] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:48.475020 18547 raft_consensus.cc:482] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:48.475039 18547 raft_consensus.cc:3037] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.474967 18162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6be27108e94908b16d946582bb7284" candidate_uuid: "587367586b4d4f3586fe10e73428a5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2f8c0d3d5741099931655457cf07c8" is_pre_election: true
I20260812 00:09:48.475142 18162 raft_consensus.cc:2454] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587367586b4d4f3586fe10e73428a5c6 in term 0.
I20260812 00:09:48.476320 18547 raft_consensus.cc:504] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.476500 18547 leader_election.cc:288] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [CANDIDATE]: Term 1 election: Requested vote from peers cd2f8c0d3d5741099931655457cf07c8 (127.16.99.193:36389), be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383)
I20260812 00:09:48.476691 18309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6be27108e94908b16d946582bb7284" candidate_uuid: "587367586b4d4f3586fe10e73428a5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6"
I20260812 00:09:48.476696 18162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b6be27108e94908b16d946582bb7284" candidate_uuid: "587367586b4d4f3586fe10e73428a5c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2f8c0d3d5741099931655457cf07c8"
I20260812 00:09:48.476787 18309 raft_consensus.cc:3037] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.476826 18162 raft_consensus.cc:3037] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:48.477483 18309 raft_consensus.cc:2454] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587367586b4d4f3586fe10e73428a5c6 in term 1.
I20260812 00:09:48.477696 18391 leader_election.cc:302] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 587367586b4d4f3586fe10e73428a5c6, be48d365ffcc4c8d90e9c3c2b2d831f6; no voters:
I20260812 00:09:48.477720 18162 raft_consensus.cc:2454] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587367586b4d4f3586fe10e73428a5c6 in term 1.
I20260812 00:09:48.477901 18547 raft_consensus.cc:2781] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:48.478124 18547 raft_consensus.cc:686] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [term 1 LEADER]: Becoming Leader. State: Replica: 587367586b4d4f3586fe10e73428a5c6, State: Running, Role: LEADER
I20260812 00:09:48.478216 18547 consensus_queue.cc:237] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } }
I20260812 00:09:48.478760 18580 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: TxnStatusTablet state changed. Reason: New leader 587367586b4d4f3586fe10e73428a5c6. Latest consensus state: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.478879 18580 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:09:48.479084 18582 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:09:48.479231 18582 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:09:48.479180 17995 catalog_manager.cc:5495] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } health_report { overall_health: HEALTHY } } }
I20260812 00:09:48.514981 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.511447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:32901 (local address 127.0.0.1:34968)
0812 00:09:48.511599 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.511611 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.511681 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.511938 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.511938 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.511940 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:48.511994 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.511998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.513013 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.513014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.513639 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.513642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.514232 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.514233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.514383 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.514419 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:48.514789 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:48.514801 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.514845 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.514916 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 00:09:48.514976 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.511540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.16.99.195:32901)
0812 00:09:48.511597 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.511600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.511685 (+ 85us) server_negotiation.cc:407] Connection header received
0812 00:09:48.511804 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.511805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.511831 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.511912 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:48.512068 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.512925 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.513741 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.514087 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.514169 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.514524 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:48.514712 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:48.514801 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.514850 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.514878 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:09:48.516595 18309 raft_consensus.cc:1261] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Refusing update from remote peer 587367586b4d4f3586fe10e73428a5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:48.516628 18162 raft_consensus.cc:1261] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Refusing update from remote peer 587367586b4d4f3586fe10e73428a5c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:48.516918 18547 consensus_queue.cc:1035] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:48.517045 18580 consensus_queue.cc:1035] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:48.517716 18546 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: TxnStatusTablet state changed. Reason: New leader 587367586b4d4f3586fe10e73428a5c6. Latest consensus state: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.517818 18546 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:48.517872 18557 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: TxnStatusTablet state changed. Reason: New leader 587367586b4d4f3586fe10e73428a5c6. Latest consensus state: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.517963 18557 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:48.519364 18552 mvcc.cc:205] Tried to move back new op lower bound from 7317476919361265664 to 7317476919207604224. Current Snapshot: MvccSnapshot[applied={T|T < 7317476919361265664 or (T in {7317476919361265664})}]
I20260812 00:09:48.519443 18588 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.519558 18588 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:09:48.519531 18557 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.519610 18557 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P cd2f8c0d3d5741099931655457cf07c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:48.519670 18547 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.519734 18547 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P 587367586b4d4f3586fe10e73428a5c6: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:09:48.520212 18546 tablet_replica.cc:439] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "587367586b4d4f3586fe10e73428a5c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2f8c0d3d5741099931655457cf07c8" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 36389 } } peers { permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 } } peers { permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 } } }
I20260812 00:09:48.520318 18546 tablet_replica.cc:442] T 5b6be27108e94908b16d946582bb7284 P be48d365ffcc4c8d90e9c3c2b2d831f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:48.526605 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.524052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:51033 (local address 127.16.99.195:32901)
0812 00:09:48.524115 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.524117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.524253 (+ 136us) server_negotiation.cc:407] Connection header received
0812 00:09:48.524322 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.524323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.524335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.524391 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:48.524511 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.525001 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.525822 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.526034 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.526122 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.526316 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:48.526381 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:48.526436 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.526499 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.526522 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:09:48.526641 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.523984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:32901 (local address 127.16.99.193:51033)
0812 00:09:48.524180 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.524196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.524270 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.524395 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.524396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.524397 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:48.524452 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.524454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.525089 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.525091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.525704 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.525705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.526126 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.526127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.526190 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.526214 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:48.526439 (+ 225us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:48.526442 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.526494 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.526554 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260812 00:09:48.529372 18463 raft_consensus.cc:1261] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Refusing update from remote peer cd2f8c0d3d5741099931655457cf07c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:48.529426 18309 raft_consensus.cc:1261] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Refusing update from remote peer cd2f8c0d3d5741099931655457cf07c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:48.529620 18557 consensus_queue.cc:1035] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "587367586b4d4f3586fe10e73428a5c6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:48.529726 18545 consensus_queue.cc:1035] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be48d365ffcc4c8d90e9c3c2b2d831f6" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:48.539191 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.536340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:36127 (local address 127.16.99.193:36389)
0812 00:09:48.536387 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.536389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.536470 (+ 81us) server_negotiation.cc:407] Connection header received
0812 00:09:48.536545 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.536546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.536555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.536602 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:48.536810 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.537302 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.538504 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.538703 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.538758 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.538980 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:48.539010 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:48.539046 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.539106 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.539127 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:09:48.539244 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.536323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:36389 (local address 127.16.99.195:36127)
0812 00:09:48.536397 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:48.536410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:48.536477 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:48.536632 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:48.536632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:48.536634 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:48.536720 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.536722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.537383 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.537384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.538409 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:48.538413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.538787 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.538788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:48.538866 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.538890 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:48.539050 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:48.539051 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:09:48.539106 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:09:48.539159 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260812 00:09:48.552279 17996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:48.552430 17996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:48.553190 17996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 782925ae0a544ffd823148ca7f1ff118: Sending DeleteTablet for 3 replicas of tablet 83f69631ac4f4f1f856bc2e0f5c9dde4
I20260812 00:09:48.553486 18141 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:48 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:37758
I20260812 00:09:48.553594 18563 tablet_replica.cc:330] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: stopping tablet replica
I20260812 00:09:48.553577 18289 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:48 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51708
I20260812 00:09:48.553658 18563 raft_consensus.cc:2229] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:09:48.553720 18563 raft_consensus.cc:2258] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:48.553800 18564 tablet_replica.cc:330] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: stopping tablet replica
I20260812 00:09:48.553910 18564 raft_consensus.cc:2229] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:48.553977 18564 raft_consensus.cc:2258] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:48.554232 18434 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:48 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34956
I20260812 00:09:48.554374 18562 tablet_replica.cc:330] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: stopping tablet replica
I20260812 00:09:48.554373 18563 ts_tablet_manager.cc:1905] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:48.554432 18562 raft_consensus.cc:2229] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:48.554478 18562 raft_consensus.cc:2258] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:48.554792 18562 ts_tablet_manager.cc:1905] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:48.556411 18563 ts_tablet_manager.cc:1918] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:48.556417 18564 ts_tablet_manager.cc:1905] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:48.556484 18563 log.cc:1196] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/83f69631ac4f4f1f856bc2e0f5c9dde4
I20260812 00:09:48.556514 18562 ts_tablet_manager.cc:1918] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:48.556564 18562 log.cc:1196] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/83f69631ac4f4f1f856bc2e0f5c9dde4
I20260812 00:09:48.556762 18563 ts_tablet_manager.cc:1939] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P cd2f8c0d3d5741099931655457cf07c8: Deleting consensus metadata
I20260812 00:09:48.556792 18562 ts_tablet_manager.cc:1939] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P 587367586b4d4f3586fe10e73428a5c6: Deleting consensus metadata
I20260812 00:09:48.557807 17981 catalog_manager.cc:4841] TS 587367586b4d4f3586fe10e73428a5c6 (127.16.99.195:32901): tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 (table test-table [id=b153a27a03bc456fb96ae82430e9ff84]) successfully deleted
I20260812 00:09:48.558279 18564 ts_tablet_manager.cc:1918] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:48.558391 18564 log.cc:1196] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/83f69631ac4f4f1f856bc2e0f5c9dde4
I20260812 00:09:48.558604 18564 ts_tablet_manager.cc:1939] T 83f69631ac4f4f1f856bc2e0f5c9dde4 P be48d365ffcc4c8d90e9c3c2b2d831f6: Deleting consensus metadata
I20260812 00:09:48.559003 17978 catalog_manager.cc:4841] TS cd2f8c0d3d5741099931655457cf07c8 (127.16.99.193:36389): tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 (table test-table [id=b153a27a03bc456fb96ae82430e9ff84]) successfully deleted
I20260812 00:09:48.559000 17979 catalog_manager.cc:4841] TS be48d365ffcc4c8d90e9c3c2b2d831f6 (127.16.99.194:40383): tablet 83f69631ac4f4f1f856bc2e0f5c9dde4 (table test-table [id=b153a27a03bc456fb96ae82430e9ff84]) successfully deleted
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[17947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:09:48.569463 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.557764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.16.99.193:36389)
0812 00:09:48.558752 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.558755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.558765 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:09:48.558800 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.558801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.558815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.558883 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:48.559731 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.560213 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.561328 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.561517 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.561575 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.565736 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:48.565752 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:48.565753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:48.565763 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:48.568166 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.568539 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.568543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.568544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.568621 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.568836 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.568837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.568837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.569034 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:48.569187 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.569245 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.569340 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957}
W20260812 00:09:48.569800 18205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37768
I20260812 00:09:48.577380 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:48.570962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.16.99.254:39391)
0812 00:09:48.571014 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:48.571017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:48.571117 (+ 100us) server_negotiation.cc:407] Connection header received
0812 00:09:48.571176 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:48.571177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:48.571188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:48.571230 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:48.571763 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.572268 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.572989 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:48.573188 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:48.573268 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:48.574667 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:48.574669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:48.574669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:48.574677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:48.576193 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.576570 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.576571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.576571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.576633 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:48.576849 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:48.576849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:48.576850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:48.576970 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:48.577074 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:48.577181 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 00:09:48.577268 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 00:09:48.577692 17996 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52076
I20260812 00:09:48.578380 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 18073
I20260812 00:09:48.589522 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 18221
I20260812 00:09:48.596814 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 18372
I20260812 00:09:48.604888 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 17961
2026-08-12T00:09:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2529 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[18603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[18603](info): set up 2 sockets
Aug 12 00:09:48 dist-test-slave-zpfg krb5kdc[18603](info): commencing operation
krb5kdc: starting...
W20260812 00:09:49.714541 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:09:49 dist-test-slave-zpfg krb5kdc[18603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:09:49.849290 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45249
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:49.934281 18618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:49.934592 18618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:49.934634 18618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:49.935855 18618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:09:49.935930 18618 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:09:49.935952 18618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:49.935976 18618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:49.936002 18618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:09:49.936033 18618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:09:49.937609 18618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:45249
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:49.937914 18618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:49.938159 18618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:49.943903 18623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:49.943892 18626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:49.943895 18624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:49.944156 18618 server_base.cc:1034] running on GCE node
I20260812 00:09:49.944583 18618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:49.944936 18618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:49.946117 18618 hybrid_clock.cc:648] HybridClock initialized: now 1786493389946103 us; error 29 us; skew 500 ppm
Aug 12 00:09:49 dist-test-slave-zpfg krb5kdc[18603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:49.948978 18618 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:09:49.949456 18618 webserver.cc:460] Webserver started at http://127.16.99.254:37599/ using document root <none> and password file <none>
I20260812 00:09:49.949700 18618 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:49.949748 18618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:49.949846 18618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:49.951036 18618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "c048e306663741ae912504f300bd10be"
format_stamp: "Formatted at 2026-08-12 00:09:49 on dist-test-slave-zpfg"
server_key: "f0b47c64d5e2dbaf52a06e33a941a04d"
server_key_iv: "6b33ce5e59513b087be54d14491398a0"
server_key_version: "encryptionkey@0"
I20260812 00:09:49.951350 18618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "c048e306663741ae912504f300bd10be"
format_stamp: "Formatted at 2026-08-12 00:09:49 on dist-test-slave-zpfg"
server_key: "f0b47c64d5e2dbaf52a06e33a941a04d"
server_key_iv: "6b33ce5e59513b087be54d14491398a0"
server_key_version: "encryptionkey@0"
I20260812 00:09:49.952750 18618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:09:49.953550 18633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:49.953752 18618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:49.953841 18618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "c048e306663741ae912504f300bd10be"
format_stamp: "Formatted at 2026-08-12 00:09:49 on dist-test-slave-zpfg"
server_key: "f0b47c64d5e2dbaf52a06e33a941a04d"
server_key_iv: "6b33ce5e59513b087be54d14491398a0"
server_key_version: "encryptionkey@0"
I20260812 00:09:49.953900 18618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:49.981048 18618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:49.983750 18618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:49.983883 18618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:49.988767 18618 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:45249
I20260812 00:09:49.988889 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:45249 every 8 connection(s)
I20260812 00:09:49.989229 18618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:09:49.990018 18700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:49.992455 18700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Bootstrap starting.
I20260812 00:09:49.993079 18700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:49.993392 18700 log.cc:824] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:49.994194 18700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: No bootstrap required, opened a new log
I20260812 00:09:49.995867 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 18618
I20260812 00:09:49.995971 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:09:49.996269 16783 external_mini_cluster.cc:1428] Setting key da9e564effc8f185788a4419836b8a67
I20260812 00:09:49.997046 18700 raft_consensus.cc:348] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c048e306663741ae912504f300bd10be" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45249 } }
I20260812 00:09:49.997200 18700 raft_consensus.cc:374] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:49.997222 18700 raft_consensus.cc:729] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c048e306663741ae912504f300bd10be, State: Initialized, Role: FOLLOWER
I20260812 00:09:49.997372 18700 consensus_queue.cc:260] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c048e306663741ae912504f300bd10be" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45249 } }
I20260812 00:09:49.997434 18700 raft_consensus.cc:388] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:09:49.997457 18700 raft_consensus.cc:482] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:09:49.997521 18700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:49.998320 18700 raft_consensus.cc:504] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c048e306663741ae912504f300bd10be" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45249 } }
I20260812 00:09:49.998483 18700 leader_election.cc:302] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c048e306663741ae912504f300bd10be; no voters:
I20260812 00:09:49.998677 18700 leader_election.cc:288] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:09:49.998703 18705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:49.998824 18705 raft_consensus.cc:686] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [term 1 LEADER]: Becoming Leader. State: Replica: c048e306663741ae912504f300bd10be, State: Running, Role: LEADER
I20260812 00:09:49.998910 18705 consensus_queue.cc:237] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c048e306663741ae912504f300bd10be" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45249 } }
I20260812 00:09:49.998965 18700 sys_catalog.cc:564] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:09:49.999277 18705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c048e306663741ae912504f300bd10be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c048e306663741ae912504f300bd10be" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45249 } } }
I20260812 00:09:49.999351 18705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:49.999472 18705 sys_catalog.cc:455] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [sys.catalog]: SysCatalogTable state changed. Reason: New leader c048e306663741ae912504f300bd10be. Latest consensus state: current_term: 1 leader_uuid: "c048e306663741ae912504f300bd10be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c048e306663741ae912504f300bd10be" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45249 } } }
I20260812 00:09:49.999543 18705 sys_catalog.cc:458] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:50.000197 18712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:09:50.000662 18712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:09:50.003537 18712 catalog_manager.cc:1347] Generated new cluster ID: bb20750257e74db5bbe6d71c9f664e16
I20260812 00:09:50.003615 18712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:50.009733 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:49.996876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.16.99.254:45249)
0812 00:09:49.997097 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:49.997106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:49.997133 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:09:49.997274 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:49.997283 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:49.997363 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:49.997528 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:49.998440 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:49.999862 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.001028 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.001596 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.001737 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.004779 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.004783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.004784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.004798 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.007761 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.008291 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.008294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.008295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.008430 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.008681 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.008682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.008682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.009006 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.009158 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.009326 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.009471 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":112,"threads_started":1}
I20260812 00:09:50.012737 18712 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:09:50.013558 18712 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:09:50.020745 18712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Generated new TSK 0
I20260812 00:09:50.020985 18712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:09:50.053103 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45249
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:50.141049 18728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:50.141283 18728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:50.141311 18728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:50.142658 18728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:50.142706 18728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:50.142729 18728 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:09:50.142741 18728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:09:50.142783 18728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:09:50.144364 18728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:45249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:50.144625 18728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:50.144886 18728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:50.150193 18733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:50.150197 18734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:50.150385 18736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:50.150614 18728 server_base.cc:1034] running on GCE node
I20260812 00:09:50.150851 18728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:50.151131 18728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:50.152313 18728 hybrid_clock.cc:648] HybridClock initialized: now 1786493390152290 us; error 42 us; skew 500 ppm
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:50.155450 18728 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:09:50.155958 18728 webserver.cc:460] Webserver started at http://127.16.99.193:43435/ using document root <none> and password file <none>
I20260812 00:09:50.156236 18728 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:50.156309 18728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:50.156459 18728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:50.157618 18728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "77089832ee2b44d6bc1cc8c775f022eb"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "0fae074bd3f47b79ab635a489c5c97cb"
server_key_iv: "c6f0f87a98e63ed2dbb7a4c564a6a188"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.157992 18728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "77089832ee2b44d6bc1cc8c775f022eb"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "0fae074bd3f47b79ab635a489c5c97cb"
server_key_iv: "c6f0f87a98e63ed2dbb7a4c564a6a188"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.159504 18728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:09:50.160385 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.160596 18728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:50.160729 18728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "77089832ee2b44d6bc1cc8c775f022eb"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "0fae074bd3f47b79ab635a489c5c97cb"
server_key_iv: "c6f0f87a98e63ed2dbb7a4c564a6a188"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.160821 18728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:50.176808 18728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:50.179512 18728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:50.179694 18728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:50.180277 18728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:50.180507 18728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.180605 18728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:50.180642 18728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:50.188899 18728 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:33301
I20260812 00:09:50.189051 18865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:33301 every 8 connection(s)
I20260812 00:09:50.189354 18728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:09:50.192384 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.181115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:51449 (local address 127.16.99.254:45249)
0812 00:09:50.181352 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.181355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.181481 (+ 126us) server_negotiation.cc:407] Connection header received
0812 00:09:50.182024 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.182026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.182043 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.182144 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.183141 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.184054 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.185871 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.186259 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.186371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.189569 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.189571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.189571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.189580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.191002 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.191423 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.191423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.191424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.191479 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.191811 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.191812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.191812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.191927 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.191983 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.192234 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.192316 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260812 00:09:50.192656 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.181096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45249 (local address 127.16.99.193:51449)
0812 00:09:50.181280 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.181383 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.181918 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.182204 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.182208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.182658 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:50.183019 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.183040 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.184168 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.184169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.185745 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.185749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.186472 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.186473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.186677 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.187293 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:50.187306 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:50.189441 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:50.191086 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.191093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.191094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.191346 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.191547 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.191548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.191548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.191658 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.191989 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:50.191994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:50.192120 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.192207 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.192358 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260812 00:09:50.193208 18866 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45249
I20260812 00:09:50.193325 18866 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:50.193516 18866 heartbeater.cc:511] Master 127.16.99.254:45249 requested a full tablet report, sending...
I20260812 00:09:50.194113 18650 ts_manager.cc:194] Registered new tserver with Master: 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301)
I20260812 00:09:50.195096 18650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:51449
I20260812 00:09:50.199704 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 18728
I20260812 00:09:50.199836 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:09:50.200021 16783 external_mini_cluster.cc:1428] Setting key 25842d61f9de515381497062b676bde1
I20260812 00:09:50.201067 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.194473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45249 (local address 127.16.99.193:45985)
0812 00:09:50.194623 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.194635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.194697 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.194823 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.194824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.195018 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:50.195096 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.195099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.195992 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.195993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.196533 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.196535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.196903 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.196904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.196970 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.197401 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:50.197406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:50.197774 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:50.199695 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.199696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.199697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.199945 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.200194 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.200194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.200195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.200262 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.200555 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:50.200556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:50.200633 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.200728 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.200894 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260812 00:09:50.202693 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.194570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:45985 (local address 127.16.99.254:45249)
0812 00:09:50.194620 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.194622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.194686 (+ 64us) server_negotiation.cc:407] Connection header received
0812 00:09:50.194754 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.194754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.194763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.194818 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.195199 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.195884 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.196621 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.196839 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.196893 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.197862 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.197864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.197864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.197871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.199573 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.200026 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.200027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.200028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.200119 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.200349 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.200350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.200351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.200478 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.200564 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.202458 (+ 1894us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.202572 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:09:50.230154 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45249
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:50.314877 18876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:50.315150 18876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:50.315198 18876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:50.316418 18876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:50.316500 18876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:50.316536 18876 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:09:50.316562 18876 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:09:50.316601 18876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:09:50.318109 18876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:45249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:50.318446 18876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:50.318703 18876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:50.324560 18882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:50.324590 18876 server_base.cc:1034] running on GCE node
W20260812 00:09:50.324560 18884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:50.324563 18881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:50.325063 18876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:50.325282 18876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:50.326411 18876 hybrid_clock.cc:648] HybridClock initialized: now 1786493390326381 us; error 42 us; skew 500 ppm
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:50.329100 18876 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:09:50.329689 18876 webserver.cc:460] Webserver started at http://127.16.99.194:43101/ using document root <none> and password file <none>
I20260812 00:09:50.329954 18876 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:50.330032 18876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:50.330186 18876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:50.331362 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "52532c19a6b34932bd87a23229dd58fd"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "278a093a9f38d632d17b9d28a419b0c1"
server_key_iv: "76239a596f421d980c6ce1b7b917d2f3"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.331749 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "52532c19a6b34932bd87a23229dd58fd"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "278a093a9f38d632d17b9d28a419b0c1"
server_key_iv: "76239a596f421d980c6ce1b7b917d2f3"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.333168 18876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:09:50.334017 18893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.334268 18876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:50.334444 18876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "52532c19a6b34932bd87a23229dd58fd"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "278a093a9f38d632d17b9d28a419b0c1"
server_key_iv: "76239a596f421d980c6ce1b7b917d2f3"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.334539 18876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:50.343776 18876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:50.345963 18876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:50.346161 18876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:50.346917 18876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:50.346987 18876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.347026 18876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:50.347052 18876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:50.355561 18876 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:33563
I20260812 00:09:50.356033 18876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:09:50.356079 19017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:33563 every 8 connection(s)
I20260812 00:09:50.358565 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.347251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:44441 (local address 127.16.99.254:45249)
0812 00:09:50.347312 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.347315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.347630 (+ 315us) server_negotiation.cc:407] Connection header received
0812 00:09:50.348142 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.348144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.348160 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.348247 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.348971 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.349786 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.351789 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.352011 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.352103 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.354955 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.354957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.354958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.354966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.356864 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.357458 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.357459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.357460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.357525 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.357808 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.357809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.357809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.357928 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.357983 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.358408 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.358495 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 00:09:50.358866 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.347254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45249 (local address 127.16.99.194:44441)
0812 00:09:50.347481 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.347523 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.348032 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.348329 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.348333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.348586 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:50.348868 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.348873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.349895 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.349896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.351659 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.351664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.352126 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.352127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.352275 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.352856 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:50.352864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:50.354821 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:50.356981 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.356987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.356988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.357340 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.357621 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.357622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.357622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.357743 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.357994 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:50.357999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:50.358145 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.358271 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.358549 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":153,"threads_started":1}
I20260812 00:09:50.359678 19018 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45249
I20260812 00:09:50.359772 19018 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:50.360006 19018 heartbeater.cc:511] Master 127.16.99.254:45249 requested a full tablet report, sending...
I20260812 00:09:50.360611 18649 ts_manager.cc:194] Registered new tserver with Master: 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563)
I20260812 00:09:50.361126 18649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:44441
I20260812 00:09:50.366317 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 18876
I20260812 00:09:50.366488 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:09:50.366688 16783 external_mini_cluster.cc:1428] Setting key 0da02310b512fc18fb51b7028e339aeb
I20260812 00:09:50.367422 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.360763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45249 (local address 127.16.99.194:36761)
0812 00:09:50.360920 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.360936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.361001 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.361206 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.361207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.361578 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:50.361660 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.361663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.362323 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.362324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.362998 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.363000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.363550 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.363551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.363620 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.364314 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:50.364319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:50.364643 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:50.366227 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.366228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.366229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.366487 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.366738 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.366739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.366739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.366817 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.367138 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:50.367139 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:50.367188 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.367252 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.367342 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260812 00:09:50.367477 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.360825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:36761 (local address 127.16.99.254:45249)
0812 00:09:50.360870 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.360872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.360994 (+ 122us) server_negotiation.cc:407] Connection header received
0812 00:09:50.361046 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.361047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.361057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.361125 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.361746 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.362206 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.363123 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.363443 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.363515 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.364730 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.364732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.364732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.364738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.366137 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.366602 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.366603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.366604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.366666 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.366881 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.366882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.366882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.367057 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.367131 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.367299 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.367380 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:09:50.395637 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45249
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:50.484896 19027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:50.485124 19027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:50.485152 19027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:50.486274 19027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:50.486312 19027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:50.486333 19027 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 00:09:50.486362 19027 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:09:50.486380 19027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:09:50.487913 19027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:45249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:50.488130 19027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:50.488318 19027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:09:50.493813 19027 server_base.cc:1034] running on GCE node
W20260812 00:09:50.493734 19037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:50.493764 19034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:50.493757 19035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:50.494165 19027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:50.494484 19027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:50.495669 19027 hybrid_clock.cc:648] HybridClock initialized: now 1786493390495674 us; error 58 us; skew 500 ppm
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:50.503502 19027 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:09:50.504050 19027 webserver.cc:460] Webserver started at http://127.16.99.195:41455/ using document root <none> and password file <none>
I20260812 00:09:50.504331 19027 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:50.504384 19027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:50.504530 19027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:50.505663 19027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "237f45e60aa94040a46795276ab1d8a2"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "38e2c43a0f781f71921c184af98635ff"
server_key_iv: "713be0f0c3838418ecdaffb414fb4e15"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.506045 19027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "237f45e60aa94040a46795276ab1d8a2"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "38e2c43a0f781f71921c184af98635ff"
server_key_iv: "713be0f0c3838418ecdaffb414fb4e15"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.507544 19027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:09:50.508494 19046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.508747 19027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:50.508855 19027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "237f45e60aa94040a46795276ab1d8a2"
format_stamp: "Formatted at 2026-08-12 00:09:50 on dist-test-slave-zpfg"
server_key: "38e2c43a0f781f71921c184af98635ff"
server_key_iv: "713be0f0c3838418ecdaffb414fb4e15"
server_key_version: "encryptionkey@0"
I20260812 00:09:50.508932 19027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:50.527878 19027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:50.530011 19027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:50.530171 19027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:50.530871 19027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:50.530922 19027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.530946 19027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:50.530958 19027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:50.538807 19027 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:45463
I20260812 00:09:50.539225 19027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:09:50.539197 19168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:45463 every 8 connection(s)
I20260812 00:09:50.541328 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.531426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:54051 (local address 127.16.99.254:45249)
0812 00:09:50.531500 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.531503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.531775 (+ 272us) server_negotiation.cc:407] Connection header received
0812 00:09:50.532274 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.532276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.532289 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.532369 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.533094 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.533659 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.534803 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.535005 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.535083 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.537824 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.537826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.537826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.537835 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.539814 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.540254 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.540255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.540256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.540317 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.540622 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.540623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.540624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.540757 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.540863 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.541147 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.541239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:09:50.541515 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.531398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45249 (local address 127.16.99.195:54051)
0812 00:09:50.531585 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.531627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.532174 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.532467 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.532470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.532792 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:50.532989 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.532994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.533768 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.533769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.534709 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.534711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.535092 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.535093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.535222 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.535774 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:50.535783 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:50.537713 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:50.539914 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.539921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.539924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.540174 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.540395 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.540395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.540396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.540542 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.540847 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:50.540854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:50.541001 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.541112 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.541280 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":117,"threads_started":1}
I20260812 00:09:50.542034 19169 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45249
I20260812 00:09:50.542152 19169 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:50.542383 19169 heartbeater.cc:511] Master 127.16.99.254:45249 requested a full tablet report, sending...
I20260812 00:09:50.542759 18649 ts_manager.cc:194] Registered new tserver with Master: 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463)
I20260812 00:09:50.543177 18649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:54051
I20260812 00:09:50.543556 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 19027
I20260812 00:09:50.543640 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:09:50.543844 16783 external_mini_cluster.cc:1428] Setting key 12c8ee102552355bb8363260d3ac1fd5
I20260812 00:09:50.544816 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:09:50.549535 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.543319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58497 (local address 127.16.99.254:45249)
0812 00:09:50.543364 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.543365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.543495 (+ 130us) server_negotiation.cc:407] Connection header received
0812 00:09:50.543560 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.543561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.543572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.543622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.543995 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.544669 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.545449 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.545658 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.545743 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.546875 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.546876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.546876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.546883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.548330 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.548712 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.548713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.548714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.548771 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.548986 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.548987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.548987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.549104 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.549181 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.549359 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.549438 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:09:50.549546 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.543261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45249 (local address 127.16.99.195:58497)
0812 00:09:50.543423 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.543433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.543511 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.543625 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.543625 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.543835 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:50.543920 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.543923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.544771 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.544772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.545364 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.545365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.545745 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.545746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.545823 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.546335 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:50.546389 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:50.546772 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:50.548435 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.548437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.548437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.548622 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.548851 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.548852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.548852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.548913 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.549172 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:50.549173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:50.549229 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.549331 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.549440 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:50.570992 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.563422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.16.99.254:45249)
0812 00:09:50.563485 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.563487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.563500 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:09:50.563548 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.563550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.563562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.563634 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.564324 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.564873 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.565622 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.565827 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.565881 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.568374 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.568376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.568376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.568383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.569834 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.570181 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.570182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.570182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.570233 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.570552 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.570553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.570553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.570668 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.570749 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.570857 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.570922 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:09:50.573091 18649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:09:50.575071 18649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:50.582770 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.577844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:45463 (local address 127.0.0.1:41524)
0812 00:09:50.578754 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.578764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.578817 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.579015 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.579016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.579017 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:50.579057 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.579059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.579933 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.579934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.581376 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.581379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.582492 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.582492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.582576 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.582581 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.582633 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.582681 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":154,"thread_start_us":69,"threads_started":1}
I20260812 00:09:50.582901 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.577754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.16.99.193:33301)
0812 00:09:50.578295 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.578299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.578308 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:09:50.578421 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.578424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.578442 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.578525 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:50.578757 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.579915 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.581486 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.582284 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.582387 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.582450 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.582730 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.582776 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":114,"threads_started":1}
I20260812 00:09:50.582907 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.577781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.16.99.194:33563)
0812 00:09:50.578508 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.578512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.578524 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:09:50.578561 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.578563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.578578 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.578662 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:50.578898 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.579980 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.581442 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.582186 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.582304 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.582429 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.582591 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.582635 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":107,"threads_started":1}
I20260812 00:09:50.583228 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.577967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.16.99.195:45463)
0812 00:09:50.578184 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.578191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.578861 (+ 670us) server_negotiation.cc:407] Connection header received
0812 00:09:50.578908 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.578910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.578924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.579017 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:50.579122 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.579827 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.581486 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.582294 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.582387 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.582450 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.583077 (+ 627us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.583127 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":111,"threads_started":1}
I20260812 00:09:50.583081 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.577651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:33301 (local address 127.0.0.1:34980)
0812 00:09:50.578001 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.578016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.578072 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.578548 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.578552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.578555 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:50.578651 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.578654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.579986 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.579987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.581377 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.581379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.582490 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.582492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.582576 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.582582 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.582633 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.582681 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":75,"thread_start_us":83,"threads_started":1}
I20260812 00:09:50.583137 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.577635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:33563 (local address 127.0.0.1:51832)
0812 00:09:50.577966 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.578014 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.578079 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.578740 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.578741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.578744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:50.578806 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.578808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.580042 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.580043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.581333 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.581336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.582308 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.582309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.582443 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.582462 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.582571 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.582635 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":94,"thread_start_us":54,"threads_started":1}
I20260812 00:09:50.584069 19098 tablet_service.cc:1467] Processing CreateTablet for tablet 04291cd219184e51ad8ffdc9d047e73d (DEFAULT_TABLE table=test-table [id=188f7094209148caae8a8082b9128bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:50.584060 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 04291cd219184e51ad8ffdc9d047e73d (DEFAULT_TABLE table=test-table [id=188f7094209148caae8a8082b9128bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:50.584437 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04291cd219184e51ad8ffdc9d047e73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.584448 19098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04291cd219184e51ad8ffdc9d047e73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.587078 18797 tablet_service.cc:1467] Processing CreateTablet for tablet 04291cd219184e51ad8ffdc9d047e73d (DEFAULT_TABLE table=test-table [id=188f7094209148caae8a8082b9128bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:50.587582 19193 tablet_bootstrap.cc:492] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Bootstrap starting.
I20260812 00:09:50.587699 18797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04291cd219184e51ad8ffdc9d047e73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.588353 19196 tablet_bootstrap.cc:492] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Bootstrap starting.
I20260812 00:09:50.588383 19193 tablet_bootstrap.cc:654] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.588749 19193 log.cc:824] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:50.589032 19196 tablet_bootstrap.cc:654] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.589321 19196 log.cc:824] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:50.589705 19193 tablet_bootstrap.cc:492] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: No bootstrap required, opened a new log
I20260812 00:09:50.589752 19193 ts_tablet_manager.cc:1397] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:09:50.590157 19196 tablet_bootstrap.cc:492] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: No bootstrap required, opened a new log
I20260812 00:09:50.590215 19196 ts_tablet_manager.cc:1397] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:50.591745 19199 tablet_bootstrap.cc:492] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Bootstrap starting.
I20260812 00:09:50.591876 19193 raft_consensus.cc:348] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.592065 19193 raft_consensus.cc:374] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.592101 19193 raft_consensus.cc:729] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.592250 19193 consensus_queue.cc:260] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.592250 19196 raft_consensus.cc:348] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.592413 19196 raft_consensus.cc:374] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.592464 19196 raft_consensus.cc:729] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52532c19a6b34932bd87a23229dd58fd, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.592548 19193 ts_tablet_manager.cc:1428] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:09:50.592556 19199 tablet_bootstrap.cc:654] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.592585 19196 consensus_queue.cc:260] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.592696 19169 heartbeater.cc:503] Master 127.16.99.254:45249 was elected leader, sending a full tablet report...
I20260812 00:09:50.592864 19196 ts_tablet_manager.cc:1428] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:09:50.593024 19018 heartbeater.cc:503] Master 127.16.99.254:45249 was elected leader, sending a full tablet report...
I20260812 00:09:50.593585 19199 log.cc:824] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:50.594511 19199 tablet_bootstrap.cc:492] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: No bootstrap required, opened a new log
I20260812 00:09:50.594574 19199 ts_tablet_manager.cc:1397] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:09:50.596305 19199 raft_consensus.cc:348] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.596446 19199 raft_consensus.cc:374] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.596470 19199 raft_consensus.cc:729] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77089832ee2b44d6bc1cc8c775f022eb, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.596623 19199 consensus_queue.cc:260] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.596943 19199 ts_tablet_manager.cc:1428] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:09:50.596982 18866 heartbeater.cc:503] Master 127.16.99.254:45249 was elected leader, sending a full tablet report...
W20260812 00:09:50.607474 19019 tablet.cc:2369] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:09:50.669847 19200 raft_consensus.cc:482] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:50.670027 19200 raft_consensus.cc:504] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.670547 19200 leader_election.cc:288] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563)
I20260812 00:09:50.677585 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.673875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:55033 (local address 127.16.99.193:33301)
0812 00:09:50.673946 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.673948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.673957 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:09:50.674001 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.674002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.674013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.674081 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:50.674402 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.675035 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.676478 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.677143 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.677236 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.677247 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.677464 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.677510 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:09:50.677726 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.670733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:33563 (local address 127.16.99.195:45033)
0812 00:09:50.673687 (+ 2954us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.673704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.673771 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.674184 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.674185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.674188 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:50.674270 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.674272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.675274 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.675275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.676480 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.676484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.677342 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.677344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.677449 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.677454 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.677517 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.677579 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2925,"mutex_wait_us":16,"thread_start_us":92,"threads_started":1}
I20260812 00:09:50.677594 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.670732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:33301 (local address 127.16.99.195:55033)
0812 00:09:50.670895 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.670909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.670972 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.674150 (+ 3178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.674151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.674154 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:50.674250 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.674251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.675130 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.675131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.676369 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.676373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.677290 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.677291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.677372 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.677377 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.677438 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.677488 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":76}
I20260812 00:09:50.677810 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.673963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:45033 (local address 127.16.99.194:33563)
0812 00:09:50.674036 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.674038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.674047 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:09:50.674076 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.674078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.674087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.674147 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:50.674402 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.675182 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.676620 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.677247 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.677322 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.677330 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.677675 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.677737 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:09:50.678145 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04291cd219184e51ad8ffdc9d047e73d" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd" is_pre_election: true
I20260812 00:09:50.678186 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04291cd219184e51ad8ffdc9d047e73d" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb" is_pre_election: true
I20260812 00:09:50.678380 18819 raft_consensus.cc:2454] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 0.
I20260812 00:09:50.678385 18970 raft_consensus.cc:2454] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 0.
I20260812 00:09:50.678623 19051 leader_election.cc:302] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:50.678797 19200 raft_consensus.cc:2781] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:50.678860 19200 raft_consensus.cc:482] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:50.678885 19200 raft_consensus.cc:3037] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.679651 19200 raft_consensus.cc:504] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.679886 19200 leader_election.cc:288] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 election: Requested vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563)
I20260812 00:09:50.679965 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04291cd219184e51ad8ffdc9d047e73d" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb"
I20260812 00:09:50.679983 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04291cd219184e51ad8ffdc9d047e73d" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd"
I20260812 00:09:50.680040 18819 raft_consensus.cc:3037] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.680099 18970 raft_consensus.cc:3037] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.681131 18819 raft_consensus.cc:2454] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 1.
I20260812 00:09:50.681279 18970 raft_consensus.cc:2454] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 1.
I20260812 00:09:50.681319 19051 leader_election.cc:302] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:50.681408 19200 raft_consensus.cc:2781] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:50.681562 19200 raft_consensus.cc:686] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 1 LEADER]: Becoming Leader. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Running, Role: LEADER
I20260812 00:09:50.681667 19200 consensus_queue.cc:237] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.682682 18650 catalog_manager.cc:5495] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } health_report { overall_health: HEALTHY } } }
W20260812 00:09:50.690272 18867 tablet.cc:2369] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 00:09:50 dist-test-slave-zpfg krb5kdc[18603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:50.726526 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.719111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45249 (local address 127.0.0.1:54834)
0812 00:09:50.719281 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.719294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.719372 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.719522 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.719523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.719873 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:50.719923 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.719925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.720624 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.720625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.721216 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.721218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.721576 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.721577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.721630 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.722218 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:50.722224 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:50.723850 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:50.725355 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.725365 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.725368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.725571 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.725777 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:50.725778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:50.725779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:50.725846 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:50.726104 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:50.726114 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:50.726160 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.726243 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.726395 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260812 00:09:50.726586 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.719281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.16.99.254:45249)
0812 00:09:50.719355 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.719357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.719368 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:09:50.719438 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.719439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.719448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.719503 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:50.720007 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.720562 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.721322 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.721499 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.721553 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.723951 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:50.723952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:50.723953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:50.723960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:50.725241 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.725651 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.725652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.725652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.725710 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:50.725915 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:50.725916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:50.725916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:50.726030 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:50.726103 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.726266 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.726448 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":46}
I20260812 00:09:50.729211 18650 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:54834:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:09:50.729480 18650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:50.731207 18797 tablet_service.cc:1467] Processing CreateTablet for tablet 355f4ea17a8f49479df7ed82ccfaaf1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3b91c8a3f50476b9f145958f5a6d4c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:09:50.731278 19098 tablet_service.cc:1467] Processing CreateTablet for tablet 355f4ea17a8f49479df7ed82ccfaaf1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3b91c8a3f50476b9f145958f5a6d4c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:09:50.731433 19098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355f4ea17a8f49479df7ed82ccfaaf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.731433 18797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355f4ea17a8f49479df7ed82ccfaaf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.732980 19193 tablet_bootstrap.cc:492] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: Bootstrap starting.
I20260812 00:09:50.732980 19199 tablet_bootstrap.cc:492] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: Bootstrap starting.
I20260812 00:09:50.733114 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 355f4ea17a8f49479df7ed82ccfaaf1e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3b91c8a3f50476b9f145958f5a6d4c3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:09:50.733260 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355f4ea17a8f49479df7ed82ccfaaf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.733486 19199 tablet_bootstrap.cc:654] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.733718 19193 tablet_bootstrap.cc:654] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.734560 19193 tablet_bootstrap.cc:492] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: No bootstrap required, opened a new log
I20260812 00:09:50.734586 19199 tablet_bootstrap.cc:492] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: No bootstrap required, opened a new log
I20260812 00:09:50.734602 19196 tablet_bootstrap.cc:492] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: Bootstrap starting.
I20260812 00:09:50.734616 19193 ts_tablet_manager.cc:1397] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:50.734639 19199 ts_tablet_manager.cc:1397] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:50.734783 19193 raft_consensus.cc:348] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.734846 19193 raft_consensus.cc:374] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.734860 19193 raft_consensus.cc:729] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.734851 19199 raft_consensus.cc:348] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.734922 19193 consensus_queue.cc:260] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.734949 19199 raft_consensus.cc:374] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.734979 19199 raft_consensus.cc:729] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77089832ee2b44d6bc1cc8c775f022eb, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.735011 19193 ts_tablet_manager.cc:1428] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.735049 19199 consensus_queue.cc:260] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.735059 19200 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.735126 19200 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:50.735158 19199 ts_tablet_manager.cc:1428] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.735203 19203 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.735298 19203 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:50.735435 19196 tablet_bootstrap.cc:654] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.736269 19196 tablet_bootstrap.cc:492] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: No bootstrap required, opened a new log
I20260812 00:09:50.736338 19196 ts_tablet_manager.cc:1397] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:50.736505 19196 raft_consensus.cc:348] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.736567 19196 raft_consensus.cc:374] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.736583 19196 raft_consensus.cc:729] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52532c19a6b34932bd87a23229dd58fd, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.736655 19196 consensus_queue.cc:260] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.736763 19196 ts_tablet_manager.cc:1428] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:09:50.736917 19201 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.737005 19201 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 00:09:50.789942 19170 tablet.cc:2369] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:09:50.824487 19200 raft_consensus.cc:482] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:50.824611 19200 raft_consensus.cc:504] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.824812 19200 leader_election.cc:288] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563)
I20260812 00:09:50.825103 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355f4ea17a8f49479df7ed82ccfaaf1e" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd" is_pre_election: true
I20260812 00:09:50.825244 18970 raft_consensus.cc:2454] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 0.
I20260812 00:09:50.825103 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355f4ea17a8f49479df7ed82ccfaaf1e" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb" is_pre_election: true
I20260812 00:09:50.825317 18819 raft_consensus.cc:2454] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 0.
I20260812 00:09:50.825492 19047 leader_election.cc:302] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 52532c19a6b34932bd87a23229dd58fd; no voters:
I20260812 00:09:50.825634 19200 raft_consensus.cc:2781] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:50.825701 19200 raft_consensus.cc:482] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:50.825730 19200 raft_consensus.cc:3037] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.826623 19200 raft_consensus.cc:504] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.826788 19200 leader_election.cc:288] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 election: Requested vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563)
I20260812 00:09:50.826972 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355f4ea17a8f49479df7ed82ccfaaf1e" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb"
I20260812 00:09:50.827073 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355f4ea17a8f49479df7ed82ccfaaf1e" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd"
I20260812 00:09:50.827174 18970 raft_consensus.cc:3037] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.827095 18819 raft_consensus.cc:3037] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.827934 18970 raft_consensus.cc:2454] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 1.
I20260812 00:09:50.827941 18819 raft_consensus.cc:2454] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 1.
I20260812 00:09:50.828119 19047 leader_election.cc:302] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 52532c19a6b34932bd87a23229dd58fd; no voters:
I20260812 00:09:50.828233 19200 raft_consensus.cc:2781] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:50.828286 19200 raft_consensus.cc:686] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [term 1 LEADER]: Becoming Leader. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Running, Role: LEADER
I20260812 00:09:50.828332 19200 consensus_queue.cc:237] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.828747 19205 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: TxnStatusTablet state changed. Reason: New leader 237f45e60aa94040a46795276ab1d8a2. Latest consensus state: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.828846 19205 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:09:50.829115 19218 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:09:50.829262 19218 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:09:50.829267 18650 catalog_manager.cc:5495] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } health_report { overall_health: HEALTHY } } }
I20260812 00:09:50.856426 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.853635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.16.99.195:45463)
0812 00:09:50.853710 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.853712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.853723 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:09:50.853794 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.853795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.853822 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.853902 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:50.854062 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.854726 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.855502 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.855707 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.855782 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.856024 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.856145 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.856240 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.856312 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.856339 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:09:50.856437 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.853547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:45463 (local address 127.0.0.1:41530)
0812 00:09:50.853630 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.853642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.853701 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.853907 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.853908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.853910 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:50.853973 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.853976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.854811 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.854813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.855401 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.855403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.855781 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.855789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.855879 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.855896 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.856247 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.856260 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.856307 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.856363 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 00:09:50.858824 18819 raft_consensus.cc:1261] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Refusing update from remote peer 237f45e60aa94040a46795276ab1d8a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:50.858826 18970 raft_consensus.cc:1261] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Refusing update from remote peer 237f45e60aa94040a46795276ab1d8a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:50.859218 19200 consensus_queue.cc:1035] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:50.859458 19205 consensus_queue.cc:1035] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:50.860903 19201 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: TxnStatusTablet state changed. Reason: New leader 237f45e60aa94040a46795276ab1d8a2. Latest consensus state: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.861032 19201 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:50.862545 19203 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: TxnStatusTablet state changed. Reason: New leader 237f45e60aa94040a46795276ab1d8a2. Latest consensus state: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.862651 19203 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:50.863533 19205 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.863648 19200 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.863754 19200 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:09:50.864071 19203 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.864158 19203 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 77089832ee2b44d6bc1cc8c775f022eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:50.864281 19225 mvcc.cc:205] Tried to move back new op lower bound from 7317476928954793984 to 7317476928833482752. Current Snapshot: MvccSnapshot[applied={T|T < 7317476928954793984}]
I20260812 00:09:50.863647 19205 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 237f45e60aa94040a46795276ab1d8a2: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:09:50.864943 19201 tablet_replica.cc:439] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } } }
I20260812 00:09:50.865031 19201 tablet_replica.cc:442] T 355f4ea17a8f49479df7ed82ccfaaf1e P 52532c19a6b34932bd87a23229dd58fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:09:50.865345 19223 mvcc.cc:205] Tried to move back new op lower bound from 7317476928954793984 to 7317476928833482752. Current Snapshot: MvccSnapshot[applied={T|T < 7317476928954793984 or (T in {7317476928954793984})}]
I20260812 00:09:50.869308 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.866727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.16.99.195:45463)
0812 00:09:50.866795 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.866797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.866806 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:09:50.866837 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.866837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.866846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.866891 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:50.867030 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.867491 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.868619 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.868812 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.868873 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.869045 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.869071 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.869110 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.869186 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.869209 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:09:50.874667 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.871966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:33293 (local address 127.16.99.195:45463)
0812 00:09:50.872002 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.872003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.872018 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:09:50.872082 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.872082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.872089 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.872130 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:50.872270 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.872726 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.873887 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.874064 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.874123 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.874388 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.874422 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.874466 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.874549 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.874572 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 00:09:50.874743 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.871903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:45463 (local address 127.16.99.195:33293)
0812 00:09:50.871980 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.871990 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.872025 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.872141 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.872142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.872143 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:50.872217 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.872219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.872809 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.872811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.873791 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.873795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.874142 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.874143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.874214 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.874235 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.874488 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.874490 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.874547 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.874602 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":4}
I20260812 00:09:50.877264 18970 raft_consensus.cc:1261] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Refusing update from remote peer 237f45e60aa94040a46795276ab1d8a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:50.877229 18819 raft_consensus.cc:1261] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Refusing update from remote peer 237f45e60aa94040a46795276ab1d8a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:50.877485 19227 consensus_queue.cc:1035] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:50.877632 19205 consensus_queue.cc:1035] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:50.879446 19221 mvcc.cc:205] Tried to move back new op lower bound from 7317476929031852032 to 7317476928233054208. Current Snapshot: MvccSnapshot[applied={T|T < 7317476929031852032}]
I20260812 00:09:50.894488 18648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:50.894626 18648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:50.895447 18648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Sending DeleteTablet for 3 replicas of tablet 04291cd219184e51ad8ffdc9d047e73d
I20260812 00:09:50.895825 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 04291cd219184e51ad8ffdc9d047e73d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:50 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51832
I20260812 00:09:50.895848 18797 tablet_service.cc:1514] Processing DeleteTablet for tablet 04291cd219184e51ad8ffdc9d047e73d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:50 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260812 00:09:50.896092 18648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 00:09:50.896189 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 04291cd219184e51ad8ffdc9d047e73d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:50 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:41524
W20260812 00:09:50.896245 18648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:50.896545 19241 tablet_replica.cc:330] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: stopping tablet replica
I20260812 00:09:50.896723 19241 raft_consensus.cc:2229] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:50.896800 19241 raft_consensus.cc:2258] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:50.896967 19240 tablet_replica.cc:330] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: stopping tablet replica
I20260812 00:09:50.896682 19242 tablet_replica.cc:330] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: stopping tablet replica
I20260812 00:09:50.897646 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 33ff6c6150804804a1237a15caa644da (DEFAULT_TABLE table=test-table [id=67ff47df0cdb4df88d93c824be7be81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:50.897646 18797 tablet_service.cc:1467] Processing CreateTablet for tablet 33ff6c6150804804a1237a15caa644da (DEFAULT_TABLE table=test-table [id=67ff47df0cdb4df88d93c824be7be81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:50.897782 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ff6c6150804804a1237a15caa644da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.897794 18797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ff6c6150804804a1237a15caa644da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.898016 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 33ff6c6150804804a1237a15caa644da (DEFAULT_TABLE table=test-table [id=67ff47df0cdb4df88d93c824be7be81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:50.898128 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ff6c6150804804a1237a15caa644da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:50.899300 19199 tablet_bootstrap.cc:492] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: Bootstrap starting.
I20260812 00:09:50.899438 19196 tablet_bootstrap.cc:492] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: Bootstrap starting.
I20260812 00:09:50.899724 19199 tablet_bootstrap.cc:654] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.899749 19193 tablet_bootstrap.cc:492] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: Bootstrap starting.
I20260812 00:09:50.899884 19196 tablet_bootstrap.cc:654] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.900202 19193 tablet_bootstrap.cc:654] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:50.897795 19241 ts_tablet_manager.cc:1905] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:50.900516 19242 raft_consensus.cc:2229] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:09:50.900651 19242 raft_consensus.cc:2258] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:50.901124 19242 ts_tablet_manager.cc:1905] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:50.902431 19240 raft_consensus.cc:2229] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:50.902526 19240 raft_consensus.cc:2258] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:50.902689 19242 ts_tablet_manager.cc:1918] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:50.902770 19242 log.cc:1196] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/04291cd219184e51ad8ffdc9d047e73d
I20260812 00:09:50.902869 19193 tablet_bootstrap.cc:492] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: No bootstrap required, opened a new log
I20260812 00:09:50.902913 19193 ts_tablet_manager.cc:1397] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:09:50.902971 19241 ts_tablet_manager.cc:1918] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:50.903040 19241 log.cc:1196] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/04291cd219184e51ad8ffdc9d047e73d
I20260812 00:09:50.903060 19240 ts_tablet_manager.cc:1905] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:50.903098 19242 ts_tablet_manager.cc:1939] T 04291cd219184e51ad8ffdc9d047e73d P 237f45e60aa94040a46795276ab1d8a2: Deleting consensus metadata
I20260812 00:09:50.903081 19193 raft_consensus.cc:348] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.903165 19193 raft_consensus.cc:374] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.903191 19193 raft_consensus.cc:729] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.903249 19193 consensus_queue.cc:260] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.903327 19241 ts_tablet_manager.cc:1939] T 04291cd219184e51ad8ffdc9d047e73d P 77089832ee2b44d6bc1cc8c775f022eb: Deleting consensus metadata
I20260812 00:09:50.903357 19193 ts_tablet_manager.cc:1428] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:50.903841 18637 catalog_manager.cc:4841] TS 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463): tablet 04291cd219184e51ad8ffdc9d047e73d (table test-table [id=188f7094209148caae8a8082b9128bb4]) successfully deleted
I20260812 00:09:50.903964 19196 tablet_bootstrap.cc:492] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: No bootstrap required, opened a new log
I20260812 00:09:50.904011 19196 ts_tablet_manager.cc:1397] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 00:09:50.904165 19196 raft_consensus.cc:348] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.904263 19196 raft_consensus.cc:374] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.904299 19196 raft_consensus.cc:729] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52532c19a6b34932bd87a23229dd58fd, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.904349 19196 consensus_queue.cc:260] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.904443 19196 ts_tablet_manager.cc:1428] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:09:50.904856 19240 ts_tablet_manager.cc:1918] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:50.904932 19240 log.cc:1196] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/04291cd219184e51ad8ffdc9d047e73d
I20260812 00:09:50.905090 18637 catalog_manager.cc:4841] TS 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301): tablet 04291cd219184e51ad8ffdc9d047e73d (table test-table [id=188f7094209148caae8a8082b9128bb4]) successfully deleted
I20260812 00:09:50.905247 19240 ts_tablet_manager.cc:1939] T 04291cd219184e51ad8ffdc9d047e73d P 52532c19a6b34932bd87a23229dd58fd: Deleting consensus metadata
I20260812 00:09:50.905746 19199 tablet_bootstrap.cc:492] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: No bootstrap required, opened a new log
I20260812 00:09:50.905727 18634 catalog_manager.cc:4841] TS 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563): tablet 04291cd219184e51ad8ffdc9d047e73d (table test-table [id=188f7094209148caae8a8082b9128bb4]) successfully deleted
I20260812 00:09:50.905817 19199 ts_tablet_manager.cc:1397] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260812 00:09:50.905961 19199 raft_consensus.cc:348] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.906042 19199 raft_consensus.cc:374] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:50.906060 19199 raft_consensus.cc:729] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77089832ee2b44d6bc1cc8c775f022eb, State: Initialized, Role: FOLLOWER
I20260812 00:09:50.906102 19199 consensus_queue.cc:260] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.906190 19199 ts_tablet_manager.cc:1428] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:09:50.925026 19201 raft_consensus.cc:482] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:50.925159 19201 raft_consensus.cc:504] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.925503 19201 leader_election.cc:288] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463)
I20260812 00:09:50.929527 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.925835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:44751 (local address 127.16.99.193:33301)
0812 00:09:50.925908 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.925910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.925992 (+ 82us) server_negotiation.cc:407] Connection header received
0812 00:09:50.926172 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.926174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.926188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.926257 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:50.926563 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.927303 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.928685 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.929124 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.929215 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.929225 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.929399 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.929449 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:09:50.929549 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.925756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:33301 (local address 127.16.99.194:44751)
0812 00:09:50.925904 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.925918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.926026 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.926270 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.926271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.926273 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:50.926450 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.926453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.927394 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.927395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.928537 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.928538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.929216 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.929218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.929327 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.929331 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.929389 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.929442 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260812 00:09:50.929625 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.926046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:60287 (local address 127.16.99.195:45463)
0812 00:09:50.926098 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.926100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.926220 (+ 120us) server_negotiation.cc:407] Connection header received
0812 00:09:50.926391 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.926392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.926405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.926465 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:50.926613 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.927128 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.928656 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.929236 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.929328 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.929341 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.929508 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.929552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:09:50.929829 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ff6c6150804804a1237a15caa644da" candidate_uuid: "52532c19a6b34932bd87a23229dd58fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb" is_pre_election: true
I20260812 00:09:50.929939 18819 raft_consensus.cc:2454] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52532c19a6b34932bd87a23229dd58fd in term 0.
I20260812 00:09:50.930115 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.925969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:45463 (local address 127.16.99.194:60287)
0812 00:09:50.926141 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.926151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.926233 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.926473 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.926474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.926476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:50.926551 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.926552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.927225 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.927226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.928532 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.928537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.929329 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.929329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.929412 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.929415 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.929959 (+ 544us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.930028 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260812 00:09:50.930140 18898 leader_election.cc:302] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52532c19a6b34932bd87a23229dd58fd, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:50.930382 19201 raft_consensus.cc:2781] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:50.930429 19201 raft_consensus.cc:482] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:50.930451 19201 raft_consensus.cc:3037] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.930521 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ff6c6150804804a1237a15caa644da" candidate_uuid: "52532c19a6b34932bd87a23229dd58fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237f45e60aa94040a46795276ab1d8a2" is_pre_election: true
I20260812 00:09:50.930672 19122 raft_consensus.cc:2454] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52532c19a6b34932bd87a23229dd58fd in term 0.
I20260812 00:09:50.931397 19201 raft_consensus.cc:504] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.931550 19201 leader_election.cc:288] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [CANDIDATE]: Term 1 election: Requested vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463)
I20260812 00:09:50.931775 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ff6c6150804804a1237a15caa644da" candidate_uuid: "52532c19a6b34932bd87a23229dd58fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb"
I20260812 00:09:50.931862 18819 raft_consensus.cc:3037] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.932533 18819 raft_consensus.cc:2454] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52532c19a6b34932bd87a23229dd58fd in term 1.
I20260812 00:09:50.932680 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ff6c6150804804a1237a15caa644da" candidate_uuid: "52532c19a6b34932bd87a23229dd58fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237f45e60aa94040a46795276ab1d8a2"
I20260812 00:09:50.932766 18898 leader_election.cc:302] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52532c19a6b34932bd87a23229dd58fd, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:50.932773 19122 raft_consensus.cc:3037] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:50.932858 19201 raft_consensus.cc:2781] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:50.933038 19201 raft_consensus.cc:686] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 1 LEADER]: Becoming Leader. State: Replica: 52532c19a6b34932bd87a23229dd58fd, State: Running, Role: LEADER
I20260812 00:09:50.933116 19201 consensus_queue.cc:237] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:50.933708 19122 raft_consensus.cc:2454] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52532c19a6b34932bd87a23229dd58fd in term 1.
I20260812 00:09:50.934454 18648 catalog_manager.cc:5495] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 52532c19a6b34932bd87a23229dd58fd (127.16.99.194). New cstate: current_term: 1 leader_uuid: "52532c19a6b34932bd87a23229dd58fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } health_report { overall_health: UNKNOWN } } }
I20260812 00:09:50.942557 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.939929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.16.99.194:33563)
0812 00:09:50.939989 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.939991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.939998 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:09:50.940043 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.940043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.940061 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.940140 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:50.940251 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.940868 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.941625 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.941916 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.941993 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.942149 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.942276 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.942384 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.942437 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.942461 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:09:50.957830 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.951181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:52837 (local address 127.16.99.195:45463)
0812 00:09:50.954427 (+ 3246us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.954430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.954442 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:09:50.954477 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.954478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.954488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.954563 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:50.954804 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.955512 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.956930 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.957247 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.957327 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.957527 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.957587 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.957670 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.957712 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.957735 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3211}
I20260812 00:09:50.957880 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.950912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:45463 (local address 127.16.99.194:52837)
0812 00:09:50.951207 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.951223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.951293 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.954589 (+ 3296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.954591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.954593 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:50.954696 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.954699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.955625 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.955627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.956769 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.956798 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.957355 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.957356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.957415 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.957437 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.957671 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.957673 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.957708 (+ 35us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.957796 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":135,"threads_started":1}
I20260812 00:09:50.960665 19122 raft_consensus.cc:1261] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Refusing update from remote peer 52532c19a6b34932bd87a23229dd58fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:50.960951 18819 raft_consensus.cc:1261] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Refusing update from remote peer 52532c19a6b34932bd87a23229dd58fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:50.960999 19248 consensus_queue.cc:1035] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:50.961174 19201 consensus_queue.cc:1035] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:50.969692 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.966817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:33563 (local address 127.16.99.195:50329)
0812 00:09:50.966915 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:50.966928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:50.966969 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:50.967166 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:50.967167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:50.967168 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:50.967214 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.967217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.968108 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.968110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.968850 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:50.968854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.969299 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.969300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:50.969356 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.969361 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.969562 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.969563 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:09:50.969594 (+ 31us) client_negotiation.cc:240] Negotiation successful
0812 00:09:50.969633 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 00:09:50.969771 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:50.966871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:50329 (local address 127.16.99.194:33563)
0812 00:09:50.967038 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:50.967039 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:50.967046 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:09:50.967067 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:50.967067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:50.967077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:50.967119 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:50.967285 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.968014 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.968925 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:50.969204 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:50.969262 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:50.969449 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:50.969497 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:50.969562 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:50.969628 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 00:09:50.969664 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
W20260812 00:09:50.970122 19223 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 00:09:52.501757 18650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:52.501871 18650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:52.503237 18650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Sending DeleteTablet for 3 replicas of tablet 33ff6c6150804804a1237a15caa644da
I20260812 00:09:52.503597 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ff6c6150804804a1237a15caa644da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51832
I20260812 00:09:52.503600 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ff6c6150804804a1237a15caa644da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:41524
I20260812 00:09:52.503659 18797 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ff6c6150804804a1237a15caa644da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260812 00:09:52.503878 19284 tablet_replica.cc:330] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: stopping tablet replica
I20260812 00:09:52.503947 19284 raft_consensus.cc:2229] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:52.503994 19284 raft_consensus.cc:2258] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:52.504245 19285 tablet_replica.cc:330] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: stopping tablet replica
I20260812 00:09:52.504315 19285 raft_consensus.cc:2229] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:52.504364 19285 raft_consensus.cc:2258] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:52.504700 19285 ts_tablet_manager.cc:1905] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:52.504720 19284 ts_tablet_manager.cc:1905] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:52.505558 19283 tablet_replica.cc:330] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: stopping tablet replica
I20260812 00:09:52.505642 19283 raft_consensus.cc:2229] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:09:52.505760 19283 raft_consensus.cc:2258] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:52.506028 19283 ts_tablet_manager.cc:1905] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:52.506844 19285 ts_tablet_manager.cc:1918] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:09:52.506884 19284 ts_tablet_manager.cc:1918] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:09:52.506911 19285 log.cc:1196] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/33ff6c6150804804a1237a15caa644da
I20260812 00:09:52.506951 19284 log.cc:1196] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/33ff6c6150804804a1237a15caa644da
I20260812 00:09:52.507207 19285 ts_tablet_manager.cc:1939] T 33ff6c6150804804a1237a15caa644da P 77089832ee2b44d6bc1cc8c775f022eb: Deleting consensus metadata
I20260812 00:09:52.507251 19284 ts_tablet_manager.cc:1939] T 33ff6c6150804804a1237a15caa644da P 237f45e60aa94040a46795276ab1d8a2: Deleting consensus metadata
I20260812 00:09:52.507546 18637 catalog_manager.cc:4841] TS 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301): tablet 33ff6c6150804804a1237a15caa644da (table test-table [id=67ff47df0cdb4df88d93c824be7be81d]) successfully deleted
I20260812 00:09:52.507727 18637 catalog_manager.cc:4841] TS 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463): tablet 33ff6c6150804804a1237a15caa644da (table test-table [id=67ff47df0cdb4df88d93c824be7be81d]) successfully deleted
I20260812 00:09:52.507889 19283 ts_tablet_manager.cc:1918] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:09:52.507948 19283 log.cc:1196] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/33ff6c6150804804a1237a15caa644da
I20260812 00:09:52.508190 19283 ts_tablet_manager.cc:1939] T 33ff6c6150804804a1237a15caa644da P 52532c19a6b34932bd87a23229dd58fd: Deleting consensus metadata
I20260812 00:09:52.508617 18634 catalog_manager.cc:4841] TS 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563): tablet 33ff6c6150804804a1237a15caa644da (table test-table [id=67ff47df0cdb4df88d93c824be7be81d]) successfully deleted
Aug 12 00:09:52 dist-test-slave-zpfg krb5kdc[18603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:09:52 dist-test-slave-zpfg krb5kdc[18603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:52.532938 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.524271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.16.99.254:45249)
0812 00:09:52.524469 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:52.524471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:52.524530 (+ 59us) server_negotiation.cc:407] Connection header received
0812 00:09:52.524601 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:52.524602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:52.524613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:52.524681 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:52.525408 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.525934 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.526823 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.527027 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.527085 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.529835 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:52.529837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:52.529838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:52.529846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:52.531639 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:52.531991 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:52.531992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:52.531992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:52.532047 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:52.532353 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:52.532354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:52.532354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:52.532488 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:52.532642 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:52.532731 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 00:09:52.532809 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":137,"threads_started":1}
I20260812 00:09:52.534905 18650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:09:52.535064 18650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:52.536734 18797 tablet_service.cc:1467] Processing CreateTablet for tablet 70af8712d8c24d769028db799eef7507 (DEFAULT_TABLE table=test-table [id=3d8f1b79ff0f4c27abe49f321fb55241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:52.536749 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 70af8712d8c24d769028db799eef7507 (DEFAULT_TABLE table=test-table [id=3d8f1b79ff0f4c27abe49f321fb55241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:52.536751 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 70af8712d8c24d769028db799eef7507 (DEFAULT_TABLE table=test-table [id=3d8f1b79ff0f4c27abe49f321fb55241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:52.536901 18797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70af8712d8c24d769028db799eef7507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:52.536901 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70af8712d8c24d769028db799eef7507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:52.536916 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70af8712d8c24d769028db799eef7507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:52.538230 19296 tablet_bootstrap.cc:492] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: Bootstrap starting.
I20260812 00:09:52.538542 19298 tablet_bootstrap.cc:492] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: Bootstrap starting.
I20260812 00:09:52.539042 19298 tablet_bootstrap.cc:654] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:52.539095 19296 tablet_bootstrap.cc:654] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:52.539402 19297 tablet_bootstrap.cc:492] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: Bootstrap starting.
I20260812 00:09:52.539734 19298 tablet_bootstrap.cc:492] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: No bootstrap required, opened a new log
I20260812 00:09:52.539806 19298 ts_tablet_manager.cc:1397] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:52.539903 19296 tablet_bootstrap.cc:492] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: No bootstrap required, opened a new log
I20260812 00:09:52.539922 19298 raft_consensus.cc:348] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.539981 19296 ts_tablet_manager.cc:1397] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:52.539996 19298 raft_consensus.cc:374] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:52.540015 19298 raft_consensus.cc:729] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52532c19a6b34932bd87a23229dd58fd, State: Initialized, Role: FOLLOWER
I20260812 00:09:52.540086 19298 consensus_queue.cc:260] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.540086 19296 raft_consensus.cc:348] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.540158 19296 raft_consensus.cc:374] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:52.540190 19296 raft_consensus.cc:729] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Initialized, Role: FOLLOWER
I20260812 00:09:52.540195 19298 ts_tablet_manager.cc:1428] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:52.540257 19296 consensus_queue.cc:260] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.540472 19297 tablet_bootstrap.cc:654] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:52.540874 19296 ts_tablet_manager.cc:1428] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:52.541328 19297 tablet_bootstrap.cc:492] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: No bootstrap required, opened a new log
I20260812 00:09:52.541385 19297 ts_tablet_manager.cc:1397] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:52.541538 19297 raft_consensus.cc:348] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.541612 19297 raft_consensus.cc:374] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:52.541626 19297 raft_consensus.cc:729] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77089832ee2b44d6bc1cc8c775f022eb, State: Initialized, Role: FOLLOWER
I20260812 00:09:52.541697 19297 consensus_queue.cc:260] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.541894 19297 ts_tablet_manager.cc:1428] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:09:52.595834 19269 raft_consensus.cc:482] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:52.595950 19269 raft_consensus.cc:504] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.596153 19269 leader_election.cc:288] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563)
I20260812 00:09:52.596608 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70af8712d8c24d769028db799eef7507" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd" is_pre_election: true
I20260812 00:09:52.596558 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70af8712d8c24d769028db799eef7507" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb" is_pre_election: true
I20260812 00:09:52.596714 18970 raft_consensus.cc:2454] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 0.
I20260812 00:09:52.596740 18819 raft_consensus.cc:2454] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 0.
I20260812 00:09:52.596973 19051 leader_election.cc:302] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 52532c19a6b34932bd87a23229dd58fd, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:52.597105 19269 raft_consensus.cc:2781] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:52.597159 19269 raft_consensus.cc:482] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:52.597179 19269 raft_consensus.cc:3037] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:52.598115 19269 raft_consensus.cc:504] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.598264 19269 leader_election.cc:288] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 election: Requested vote from peers 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301), 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563)
I20260812 00:09:52.598502 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70af8712d8c24d769028db799eef7507" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd"
I20260812 00:09:52.598502 18819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70af8712d8c24d769028db799eef7507" candidate_uuid: "237f45e60aa94040a46795276ab1d8a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77089832ee2b44d6bc1cc8c775f022eb"
I20260812 00:09:52.598593 18819 raft_consensus.cc:3037] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:52.598593 18970 raft_consensus.cc:3037] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:52.599418 18819 raft_consensus.cc:2454] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 1.
I20260812 00:09:52.599418 18970 raft_consensus.cc:2454] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237f45e60aa94040a46795276ab1d8a2 in term 1.
I20260812 00:09:52.599658 19051 leader_election.cc:302] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:52.599781 19269 raft_consensus.cc:2781] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:52.599864 19269 raft_consensus.cc:686] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 1 LEADER]: Becoming Leader. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Running, Role: LEADER
I20260812 00:09:52.599918 19269 consensus_queue.cc:237] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.600571 18648 catalog_manager.cc:5495] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "237f45e60aa94040a46795276ab1d8a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } health_report { overall_health: HEALTHY } } }
I20260812 00:09:52.621675 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.619199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.16.99.195:45463)
0812 00:09:52.619377 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:52.619378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:52.619384 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:09:52.619403 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:52.619403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:52.619412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:52.619486 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:52.619643 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.620139 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.620937 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.621156 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.621236 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.621391 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:52.621427 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:52.621494 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:52.621556 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 00:09:52.621580 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260812 00:09:52.624555 18970 raft_consensus.cc:1261] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Refusing update from remote peer 237f45e60aa94040a46795276ab1d8a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:52.624583 18819 raft_consensus.cc:1261] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Refusing update from remote peer 237f45e60aa94040a46795276ab1d8a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:52.624825 19205 consensus_queue.cc:1035] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:52.624948 19269 consensus_queue.cc:1035] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:52.634971 18648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:52.635093 18648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:52.635773 18648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Sending DeleteTablet for 3 replicas of tablet 70af8712d8c24d769028db799eef7507
I20260812 00:09:52.636032 18797 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af8712d8c24d769028db799eef7507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260812 00:09:52.636047 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af8712d8c24d769028db799eef7507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51832
I20260812 00:09:52.636042 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 70af8712d8c24d769028db799eef7507 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:41524
I20260812 00:09:52.636160 19285 tablet_replica.cc:330] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: stopping tablet replica
I20260812 00:09:52.636211 19283 tablet_replica.cc:330] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: stopping tablet replica
I20260812 00:09:52.636266 19285 raft_consensus.cc:2229] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:52.636273 19283 raft_consensus.cc:2229] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:52.636318 19283 raft_consensus.cc:2258] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:52.636323 19285 raft_consensus.cc:2258] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:52.636400 19284 tablet_replica.cc:330] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: stopping tablet replica
I20260812 00:09:52.636471 19284 raft_consensus.cc:2229] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:09:52.636538 19284 raft_consensus.cc:2258] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:52.636482 18648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:09:52.636616 18648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:09:52.636927 19284 ts_tablet_manager.cc:1905] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:52.637032 19283 ts_tablet_manager.cc:1905] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:52.637790 18948 tablet_service.cc:1467] Processing CreateTablet for tablet 8a723bbd09c44a14ab1fc085139f4982 (DEFAULT_TABLE table=test-table [id=a11e5ba3ba654899b5aa243a6ec64df8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:52.637900 18948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a723bbd09c44a14ab1fc085139f4982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:52.638695 19285 ts_tablet_manager.cc:1905] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:52.638815 19283 ts_tablet_manager.cc:1918] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:52.638861 19283 log.cc:1196] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/70af8712d8c24d769028db799eef7507
I20260812 00:09:52.639038 19298 tablet_bootstrap.cc:492] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: Bootstrap starting.
I20260812 00:09:52.639098 19283 ts_tablet_manager.cc:1939] T 70af8712d8c24d769028db799eef7507 P 52532c19a6b34932bd87a23229dd58fd: Deleting consensus metadata
I20260812 00:09:52.639376 19298 tablet_bootstrap.cc:654] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:52.639410 19284 ts_tablet_manager.cc:1918] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:52.639461 19284 log.cc:1196] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/70af8712d8c24d769028db799eef7507
I20260812 00:09:52.639674 19284 ts_tablet_manager.cc:1939] T 70af8712d8c24d769028db799eef7507 P 237f45e60aa94040a46795276ab1d8a2: Deleting consensus metadata
I20260812 00:09:52.639729 18634 catalog_manager.cc:4841] TS 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563): tablet 70af8712d8c24d769028db799eef7507 (table test-table [id=3d8f1b79ff0f4c27abe49f321fb55241]) successfully deleted
I20260812 00:09:52.639839 19097 tablet_service.cc:1467] Processing CreateTablet for tablet 8a723bbd09c44a14ab1fc085139f4982 (DEFAULT_TABLE table=test-table [id=a11e5ba3ba654899b5aa243a6ec64df8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:52.639950 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a723bbd09c44a14ab1fc085139f4982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:52.640010 18637 catalog_manager.cc:4841] TS 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463): tablet 70af8712d8c24d769028db799eef7507 (table test-table [id=3d8f1b79ff0f4c27abe49f321fb55241]) successfully deleted
I20260812 00:09:52.640288 19285 ts_tablet_manager.cc:1918] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:09:52.640329 19285 log.cc:1196] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/70af8712d8c24d769028db799eef7507
I20260812 00:09:52.640484 19285 ts_tablet_manager.cc:1939] T 70af8712d8c24d769028db799eef7507 P 77089832ee2b44d6bc1cc8c775f022eb: Deleting consensus metadata
I20260812 00:09:52.640774 18797 tablet_service.cc:1467] Processing CreateTablet for tablet 8a723bbd09c44a14ab1fc085139f4982 (DEFAULT_TABLE table=test-table [id=a11e5ba3ba654899b5aa243a6ec64df8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:09:52.640878 18797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a723bbd09c44a14ab1fc085139f4982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:52.640877 18637 catalog_manager.cc:4841] TS 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301): tablet 70af8712d8c24d769028db799eef7507 (table test-table [id=3d8f1b79ff0f4c27abe49f321fb55241]) successfully deleted
I20260812 00:09:52.640884 19296 tablet_bootstrap.cc:492] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: Bootstrap starting.
I20260812 00:09:52.641096 19298 tablet_bootstrap.cc:492] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: No bootstrap required, opened a new log
I20260812 00:09:52.641141 19298 ts_tablet_manager.cc:1397] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:52.641192 19296 tablet_bootstrap.cc:654] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:52.641338 19298 raft_consensus.cc:348] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.641412 19298 raft_consensus.cc:374] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:52.641436 19298 raft_consensus.cc:729] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52532c19a6b34932bd87a23229dd58fd, State: Initialized, Role: FOLLOWER
I20260812 00:09:52.641486 19298 consensus_queue.cc:260] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.641590 19298 ts_tablet_manager.cc:1428] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:09:52.641798 19296 tablet_bootstrap.cc:492] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: No bootstrap required, opened a new log
I20260812 00:09:52.641837 19296 ts_tablet_manager.cc:1397] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:52.641932 19297 tablet_bootstrap.cc:492] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb: Bootstrap starting.
I20260812 00:09:52.641983 19296 raft_consensus.cc:348] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.642047 19296 raft_consensus.cc:374] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:52.642073 19296 raft_consensus.cc:729] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237f45e60aa94040a46795276ab1d8a2, State: Initialized, Role: FOLLOWER
I20260812 00:09:52.642118 19296 consensus_queue.cc:260] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.642211 19296 ts_tablet_manager.cc:1428] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:52.642678 19297 tablet_bootstrap.cc:654] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:52.643509 19297 tablet_bootstrap.cc:492] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb: No bootstrap required, opened a new log
I20260812 00:09:52.643589 19297 ts_tablet_manager.cc:1397] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:09:52.643738 19297 raft_consensus.cc:348] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.643810 19297 raft_consensus.cc:374] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:52.643842 19297 raft_consensus.cc:729] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77089832ee2b44d6bc1cc8c775f022eb, State: Initialized, Role: FOLLOWER
I20260812 00:09:52.643896 19297 consensus_queue.cc:260] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.643990 19297 ts_tablet_manager.cc:1428] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:52.792685 19302 raft_consensus.cc:482] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:09:52.792824 19302 raft_consensus.cc:504] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.793208 19302 leader_election.cc:288] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563), 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463)
I20260812 00:09:52.796509 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.793556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:48833 (local address 127.16.99.195:45463)
0812 00:09:52.793606 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:52.793608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:52.793635 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:09:52.793707 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:52.793708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:52.793717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:52.793772 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:52.793956 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.794661 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.795737 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.796134 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.796197 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.796205 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:52.796380 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 00:09:52.796434 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:09:52.796559 19314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.793449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:43535 (local address 127.16.99.194:33563)
0812 00:09:52.793565 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:52.793567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:52.793679 (+ 112us) server_negotiation.cc:407] Connection header received
0812 00:09:52.793827 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:52.793828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:52.793843 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:52.793931 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:09:52.794111 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.794697 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.795732 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.796161 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.796269 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.796279 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:52.796441 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 00:09:52.796486 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260812 00:09:52.796545 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.793382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:45463 (local address 127.16.99.193:48833)
0812 00:09:52.793570 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:52.793583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:52.793633 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:52.793789 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:52.793789 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:52.793793 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:52.793881 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.793883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.794746 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.794748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.795630 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.795633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.796218 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.796219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.796305 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.796309 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:09:52.796368 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:09:52.796427 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":4,"thread_start_us":131,"threads_started":1}
I20260812 00:09:52.796558 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.793384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:33563 (local address 127.16.99.193:43535)
0812 00:09:52.793581 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:52.793595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:52.793693 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:52.793958 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:52.793959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:52.793961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:09:52.794034 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.794036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.794828 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.794829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.795629 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.795631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.796283 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.796284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.796358 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.796364 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:09:52.796423 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 00:09:52.796463 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260812 00:09:52.797017 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a723bbd09c44a14ab1fc085139f4982" candidate_uuid: "77089832ee2b44d6bc1cc8c775f022eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237f45e60aa94040a46795276ab1d8a2" is_pre_election: true
I20260812 00:09:52.797128 19122 raft_consensus.cc:2454] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77089832ee2b44d6bc1cc8c775f022eb in term 0.
I20260812 00:09:52.797103 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a723bbd09c44a14ab1fc085139f4982" candidate_uuid: "77089832ee2b44d6bc1cc8c775f022eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd" is_pre_election: true
I20260812 00:09:52.797205 18970 raft_consensus.cc:2454] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77089832ee2b44d6bc1cc8c775f022eb in term 0.
I20260812 00:09:52.797345 18749 leader_election.cc:302] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:52.797528 19302 raft_consensus.cc:2781] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:09:52.797569 19302 raft_consensus.cc:482] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:09:52.797583 19302 raft_consensus.cc:3037] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:52.798504 19302 raft_consensus.cc:504] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.798661 19302 leader_election.cc:288] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [CANDIDATE]: Term 1 election: Requested vote from peers 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563), 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463)
I20260812 00:09:52.798833 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a723bbd09c44a14ab1fc085139f4982" candidate_uuid: "77089832ee2b44d6bc1cc8c775f022eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237f45e60aa94040a46795276ab1d8a2"
I20260812 00:09:52.798938 19122 raft_consensus.cc:3037] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:52.798862 18970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a723bbd09c44a14ab1fc085139f4982" candidate_uuid: "77089832ee2b44d6bc1cc8c775f022eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52532c19a6b34932bd87a23229dd58fd"
I20260812 00:09:52.799011 18970 raft_consensus.cc:3037] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:52.799721 19122 raft_consensus.cc:2454] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77089832ee2b44d6bc1cc8c775f022eb in term 1.
I20260812 00:09:52.799924 18970 raft_consensus.cc:2454] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77089832ee2b44d6bc1cc8c775f022eb in term 1.
I20260812 00:09:52.799930 18749 leader_election.cc:302] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237f45e60aa94040a46795276ab1d8a2, 77089832ee2b44d6bc1cc8c775f022eb; no voters:
I20260812 00:09:52.800069 19302 raft_consensus.cc:2781] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:52.800263 19302 raft_consensus.cc:686] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [term 1 LEADER]: Becoming Leader. State: Replica: 77089832ee2b44d6bc1cc8c775f022eb, State: Running, Role: LEADER
I20260812 00:09:52.800356 19302 consensus_queue.cc:237] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } }
I20260812 00:09:52.801100 18648 catalog_manager.cc:5495] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193). New cstate: current_term: 1 leader_uuid: "77089832ee2b44d6bc1cc8c775f022eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77089832ee2b44d6bc1cc8c775f022eb" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 33301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 } health_report { overall_health: UNKNOWN } } }
I20260812 00:09:52.828600 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.826055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.16.99.193:33301)
0812 00:09:52.826266 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:52.826267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:52.826274 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:09:52.826315 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:52.826316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:52.826377 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:52.826466 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:52.826578 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.827072 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.827840 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.828040 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.828093 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.828242 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:52.828352 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:52.828404 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:52.828503 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 00:09:52.828522 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":145,"threads_started":1}
I20260812 00:09:52.833820 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.831366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:37677 (local address 127.16.99.195:45463)
0812 00:09:52.831427 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:52.831429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:52.831552 (+ 123us) server_negotiation.cc:407] Connection header received
0812 00:09:52.831584 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:52.831585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:52.831594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:52.831640 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:52.831901 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.832425 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.833189 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.833388 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.833458 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.833618 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:52.833646 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:52.833695 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:52.833738 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 00:09:52.833759 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:09:52.833846 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.831299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:45463 (local address 127.16.99.193:37677)
0812 00:09:52.831481 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:52.831492 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:52.831547 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:52.831761 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:52.831763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:52.831765 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:52.831827 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.831828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.832503 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.832504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.833095 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.833097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.833478 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.833479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.833524 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.833539 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:52.833702 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:52.833703 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:09:52.833737 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 00:09:52.833771 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260812 00:09:52.836516 19122 raft_consensus.cc:1261] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Refusing update from remote peer 77089832ee2b44d6bc1cc8c775f022eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:52.836516 18970 raft_consensus.cc:1261] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Refusing update from remote peer 77089832ee2b44d6bc1cc8c775f022eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:09:52.836766 19302 consensus_queue.cc:1035] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52532c19a6b34932bd87a23229dd58fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:52.836928 19316 consensus_queue.cc:1035] T 8a723bbd09c44a14ab1fc085139f4982 P 77089832ee2b44d6bc1cc8c775f022eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "237f45e60aa94040a46795276ab1d8a2" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:09:52.844571 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.841874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:53045 (local address 127.16.99.193:33301)
0812 00:09:52.841918 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:52.841920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:52.842000 (+ 80us) server_negotiation.cc:407] Connection header received
0812 00:09:52.842047 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:52.842048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:52.842057 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:52.842096 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:09:52.842231 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.842836 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.843839 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.844088 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.844144 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.844373 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:09:52.844401 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:09:52.844442 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:52.844495 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 00:09:52.844515 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:09:52.844592 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:52.841797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:33301 (local address 127.16.99.195:53045)
0812 00:09:52.841936 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:52.841947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:52.842007 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:52.842105 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:52.842105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:52.842107 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:09:52.842174 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.842175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.842918 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.842920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.843746 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:52.843750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:52.844227 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:52.844228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:52.844273 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:52.844278 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:09:52.844455 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:09:52.844456 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:09:52.844487 (+ 31us) client_negotiation.cc:240] Negotiation successful
0812 00:09:52.844528 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
W20260812 00:09:52.845059 19262 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 00:09:54.371294 18650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:54.371423 18650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:09:54.372414 18650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c048e306663741ae912504f300bd10be: Sending DeleteTablet for 3 replicas of tablet 8a723bbd09c44a14ab1fc085139f4982
I20260812 00:09:54.372759 18797 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a723bbd09c44a14ab1fc085139f4982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:54 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34980
I20260812 00:09:54.372809 18948 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a723bbd09c44a14ab1fc085139f4982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:54 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51832
I20260812 00:09:54.372807 19097 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a723bbd09c44a14ab1fc085139f4982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:09:54 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:41524
I20260812 00:09:54.373051 19356 tablet_replica.cc:330] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: stopping tablet replica
I20260812 00:09:54.373127 19356 raft_consensus.cc:2229] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:54.373138 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 18728
I20260812 00:09:54.373186 19356 raft_consensus.cc:2258] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:54.374118 19356 ts_tablet_manager.cc:1905] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:54.376505 19356 ts_tablet_manager.cc:1918] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:09:54.376586 19356 log.cc:1196] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/8a723bbd09c44a14ab1fc085139f4982
I20260812 00:09:54.376818 19356 ts_tablet_manager.cc:1939] T 8a723bbd09c44a14ab1fc085139f4982 P 237f45e60aa94040a46795276ab1d8a2: Deleting consensus metadata
I20260812 00:09:54.377578 18637 catalog_manager.cc:4841] TS 237f45e60aa94040a46795276ab1d8a2 (127.16.99.195:45463): tablet 8a723bbd09c44a14ab1fc085139f4982 (table test-table [id=a11e5ba3ba654899b5aa243a6ec64df8]) successfully deleted
I20260812 00:09:54.378023 19355 tablet_replica.cc:330] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: stopping tablet replica
I20260812 00:09:54.378093 19355 raft_consensus.cc:2229] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:09:54.378129 19355 raft_consensus.cc:2258] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:09:54.378497 19355 ts_tablet_manager.cc:1905] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:09:54.380502 19355 ts_tablet_manager.cc:1918] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 00:09:54.380604 19355 log.cc:1196] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/8a723bbd09c44a14ab1fc085139f4982
I20260812 00:09:54.380892 19355 ts_tablet_manager.cc:1939] T 8a723bbd09c44a14ab1fc085139f4982 P 52532c19a6b34932bd87a23229dd58fd: Deleting consensus metadata
I20260812 00:09:54.381385 18634 catalog_manager.cc:4841] TS 52532c19a6b34932bd87a23229dd58fd (127.16.99.194:33563): tablet 8a723bbd09c44a14ab1fc085139f4982 (table test-table [id=a11e5ba3ba654899b5aa243a6ec64df8]) successfully deleted
W20260812 00:09:54.384033 18637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:33301 (error 108)
I20260812 00:09:54.384353 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 18876
I20260812 00:09:54.387671 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:54.387180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:33301 (local address 127.0.0.1:35002)
0812 00:09:54.387485 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:54.387556 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":193,"threads_started":1}
W20260812 00:09:54.387914 18637 catalog_manager.cc:4584] TS 77089832ee2b44d6bc1cc8c775f022eb (127.16.99.193:33301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a723bbd09c44a14ab1fc085139f4982: Network error: Client connection negotiation failed: client connection to 127.16.99.193:33301: connect: Connection refused (error 111)
I20260812 00:09:54.392590 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 19027
I20260812 00:09:54.402253 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 18618
2026-08-12T00:09:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5798 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:09:54 dist-test-slave-zpfg krb5kdc[19362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:09:54 dist-test-slave-zpfg krb5kdc[19362](info): set up 2 sockets
Aug 12 00:09:54 dist-test-slave-zpfg krb5kdc[19362](info): commencing operation
krb5kdc: starting...
W20260812 00:09:55.571768 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:09:55 dist-test-slave-zpfg krb5kdc[19362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:09:55.706718 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33149
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:33149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:55.802832 19375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:55.803113 19375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:55.803161 19375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:55.804488 19375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:09:55.804577 19375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:55.804605 19375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:55.804638 19375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:09:55.804672 19375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:09:55.806505 19375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:33149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:55.806865 19375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:55.807152 19375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:55.812990 19380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:55.813014 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:55.812990 19381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:55.813086 19375 server_base.cc:1034] running on GCE node
I20260812 00:09:55.813560 19375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:55.813935 19375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:55.815227 19375 hybrid_clock.cc:648] HybridClock initialized: now 1786493395815195 us; error 43 us; skew 500 ppm
Aug 12 00:09:55 dist-test-slave-zpfg krb5kdc[19362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:55.827445 19375 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:09:55.827982 19375 webserver.cc:460] Webserver started at http://127.16.99.254:45107/ using document root <none> and password file <none>
I20260812 00:09:55.828258 19375 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:55.828344 19375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:55.828514 19375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:55.829684 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "276beefc0ab24290bd63f79d23bf57e1"
format_stamp: "Formatted at 2026-08-12 00:09:55 on dist-test-slave-zpfg"
server_key: "c0e213d628ae92b4ad47b21906144070"
server_key_iv: "94e2bf779ba29154728b816ebcd0fc4a"
server_key_version: "encryptionkey@0"
I20260812 00:09:55.830066 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "276beefc0ab24290bd63f79d23bf57e1"
format_stamp: "Formatted at 2026-08-12 00:09:55 on dist-test-slave-zpfg"
server_key: "c0e213d628ae92b4ad47b21906144070"
server_key_iv: "94e2bf779ba29154728b816ebcd0fc4a"
server_key_version: "encryptionkey@0"
I20260812 00:09:55.832131 19375 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260812 00:09:55.833003 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:55.833171 19375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:55.833272 19375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "276beefc0ab24290bd63f79d23bf57e1"
format_stamp: "Formatted at 2026-08-12 00:09:55 on dist-test-slave-zpfg"
server_key: "c0e213d628ae92b4ad47b21906144070"
server_key_iv: "94e2bf779ba29154728b816ebcd0fc4a"
server_key_version: "encryptionkey@0"
I20260812 00:09:55.833357 19375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:55.843634 19375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:55.846144 19375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:55.846269 19375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:55.850555 19375 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:33149
I20260812 00:09:55.850656 19446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:33149 every 8 connection(s)
I20260812 00:09:55.851015 19375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:09:55.851948 19447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:55.853477 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 19375
I20260812 00:09:55.853580 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:09:55.853767 16783 external_mini_cluster.cc:1428] Setting key eac839fc0284b89e876d98332c3e6a5a
I20260812 00:09:55.854930 19447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1: Bootstrap starting.
I20260812 00:09:55.855631 19447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:55.856006 19447 log.cc:824] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:55.856913 19447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1: No bootstrap required, opened a new log
I20260812 00:09:55.859287 19447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276beefc0ab24290bd63f79d23bf57e1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33149 } }
I20260812 00:09:55.859508 19447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:55.859539 19447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276beefc0ab24290bd63f79d23bf57e1, State: Initialized, Role: FOLLOWER
I20260812 00:09:55.859700 19447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276beefc0ab24290bd63f79d23bf57e1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33149 } }
I20260812 00:09:55.859824 19447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:09:55.859889 19447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:09:55.859942 19447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:55.860914 19447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276beefc0ab24290bd63f79d23bf57e1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33149 } }
I20260812 00:09:55.861055 19447 leader_election.cc:302] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276beefc0ab24290bd63f79d23bf57e1; no voters:
I20260812 00:09:55.861275 19447 leader_election.cc:288] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:09:55.861363 19453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:55.861531 19453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [term 1 LEADER]: Becoming Leader. State: Replica: 276beefc0ab24290bd63f79d23bf57e1, State: Running, Role: LEADER
I20260812 00:09:55.861593 19447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:09:55.861665 19453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276beefc0ab24290bd63f79d23bf57e1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33149 } }
Aug 12 00:09:55 dist-test-slave-zpfg krb5kdc[19362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:55.862036 19453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276beefc0ab24290bd63f79d23bf57e1. Latest consensus state: current_term: 1 leader_uuid: "276beefc0ab24290bd63f79d23bf57e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276beefc0ab24290bd63f79d23bf57e1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33149 } } }
I20260812 00:09:55.862118 19453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:55.862403 19454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "276beefc0ab24290bd63f79d23bf57e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276beefc0ab24290bd63f79d23bf57e1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33149 } } }
I20260812 00:09:55.862483 19454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:55.862622 19462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:09:55.864037 19462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:09:55.865985 19462 catalog_manager.cc:1347] Generated new cluster ID: 98ec1fe36a4b4d0aa83fb395a455f6a1
I20260812 00:09:55.866050 19462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:09:55.867041 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:55.854377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.16.99.254:33149)
0812 00:09:55.854579 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:55.854589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:55.854623 (+ 34us) server_negotiation.cc:407] Connection header received
0812 00:09:55.855120 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:55.855132 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:55.855277 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:55.855401 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:55.856313 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:55.857469 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:55.858605 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:55.859044 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:55.859149 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:55.862517 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:55.862522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:55.862522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:55.862537 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:55.864990 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:55.865486 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:55.865490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:55.865490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:55.865608 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:55.865859 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:55.865861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:55.865861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:55.866187 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:55.866374 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:55.866662 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 00:09:55.866820 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
I20260812 00:09:55.895493 19462 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:09:55.896396 19462 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:09:55.901535 19462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 276beefc0ab24290bd63f79d23bf57e1: Generated new TSK 0
I20260812 00:09:55.901791 19462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:09:55.939781 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33149
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:56.028146 19474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:56.028472 19474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:56.028519 19474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:56.029799 19474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:56.029882 19474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:56.029932 19474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:09:56.031601 19474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:56.031911 19474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:56.032181 19474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:56.038219 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:56.038276 19474 server_base.cc:1034] running on GCE node
W20260812 00:09:56.038240 19484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:56.038226 19482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:56.038715 19474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:56.039054 19474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:56.040239 19474 hybrid_clock.cc:648] HybridClock initialized: now 1786493396040202 us; error 58 us; skew 500 ppm
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:56.043242 19474 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:09:56.043737 19474 webserver.cc:460] Webserver started at http://127.16.99.193:43035/ using document root <none> and password file <none>
I20260812 00:09:56.044000 19474 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:56.044075 19474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:56.044230 19474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:56.045387 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "ce4846bee5df41bd9d2aa51c289ca740"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "ec06fe38266c399eb0fd47479604769a"
server_key_iv: "4a0392d5ae38a795b158d5b4ed97c4f5"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.045780 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "ce4846bee5df41bd9d2aa51c289ca740"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "ec06fe38266c399eb0fd47479604769a"
server_key_iv: "4a0392d5ae38a795b158d5b4ed97c4f5"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.047374 19474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:09:56.048451 19491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.048847 19474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:56.048967 19474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "ce4846bee5df41bd9d2aa51c289ca740"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "ec06fe38266c399eb0fd47479604769a"
server_key_iv: "4a0392d5ae38a795b158d5b4ed97c4f5"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.049170 19474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:56.071663 19474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:56.074261 19474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:56.074477 19474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:56.074785 19474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:56.075141 19474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:56.075201 19474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.075238 19474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:56.075287 19474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.081774 19474 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:37681
I20260812 00:09:56.081841 19614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:37681 every 8 connection(s)
I20260812 00:09:56.082182 19474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:09:56.087096 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 19474
I20260812 00:09:56.087227 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:09:56.087430 16783 external_mini_cluster.cc:1428] Setting key c62cd4120c4613b49ad76d6dbc2e5cb0
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:56.094120 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:56.083005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:43471 (local address 127.16.99.254:33149)
0812 00:09:56.083083 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:56.083086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:56.083464 (+ 378us) server_negotiation.cc:407] Connection header received
0812 00:09:56.084024 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:56.084025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:56.084036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:56.084113 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:56.085738 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.086691 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.088202 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.088428 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.088519 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:56.090934 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:56.090936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:56.090936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:56.090945 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:56.092586 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:56.093042 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:56.093044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:56.093044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:56.093112 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:56.093393 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:56.093394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:56.093395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:56.093528 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:56.093601 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:56.093904 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 00:09:56.094020 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:09:56.094323 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:56.082990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33149 (local address 127.16.99.193:43471)
0812 00:09:56.083307 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:56.083349 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:56.083919 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:56.084415 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:56.084420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:56.085320 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:56.085633 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:56.085641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.086798 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.086800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:56.088107 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:56.088111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.088535 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.088536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:56.088687 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:56.089308 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:56.089317 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:56.090812 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:56.092678 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:56.092689 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:56.092692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:56.092958 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:56.093184 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:56.093185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:56.093185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:56.093317 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:56.093602 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:56.093611 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:56.093771 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 00:09:56.093884 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:09:56.094058 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260812 00:09:56.094969 19615 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33149
I20260812 00:09:56.095100 19615 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:56.095415 19615 heartbeater.cc:511] Master 127.16.99.254:33149 requested a full tablet report, sending...
I20260812 00:09:56.096029 19409 ts_manager.cc:194] Registered new tserver with Master: ce4846bee5df41bd9d2aa51c289ca740 (127.16.99.193:37681)
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
I20260812 00:09:56.096778 19409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:43471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:09:56.119338 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33149
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:56.213337 19621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:56.213579 19621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:56.213608 19621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:56.214959 19621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:56.215013 19621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:56.215041 19621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:09:56.216622 19621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:56.216897 19621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:56.217175 19621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:56.222780 19629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:56.222893 19631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:56.222822 19628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:56.223086 19621 server_base.cc:1034] running on GCE node
I20260812 00:09:56.223420 19621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:56.223707 19621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:56.224867 19621 hybrid_clock.cc:648] HybridClock initialized: now 1786493396224842 us; error 45 us; skew 500 ppm
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:56.227970 19621 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:09:56.228536 19621 webserver.cc:460] Webserver started at http://127.16.99.194:43225/ using document root <none> and password file <none>
I20260812 00:09:56.228817 19621 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:56.228870 19621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:56.229004 19621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:56.230180 19621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "3d6df94c55634ba7b6231bbdb0392890"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "545995ac10649be88bac2820c3d28d99"
server_key_iv: "1b23e411e648a0dbdec1181a340634bc"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.230623 19621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "3d6df94c55634ba7b6231bbdb0392890"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "545995ac10649be88bac2820c3d28d99"
server_key_iv: "1b23e411e648a0dbdec1181a340634bc"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.232177 19621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:09:56.233126 19638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.233351 19621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:56.233433 19621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "3d6df94c55634ba7b6231bbdb0392890"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "545995ac10649be88bac2820c3d28d99"
server_key_iv: "1b23e411e648a0dbdec1181a340634bc"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.233534 19621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:56.243642 19621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:56.246502 19621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:56.246649 19621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:56.246968 19621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:56.247370 19621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:56.247408 19621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.247433 19621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:56.247447 19621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.254724 19621 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:41595
I20260812 00:09:56.254765 19762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:41595 every 8 connection(s)
I20260812 00:09:56.255126 19621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:09:56.255934 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 19621
I20260812 00:09:56.256047 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:09:56.256265 16783 external_mini_cluster.cc:1428] Setting key 7e73bf863a4eb1c2a186020ae9f8a7b3
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
I20260812 00:09:56.266637 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:56.256047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:60341 (local address 127.16.99.254:33149)
0812 00:09:56.256128 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:56.256131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:56.256418 (+ 287us) server_negotiation.cc:407] Connection header received
0812 00:09:56.257121 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:56.257122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:56.257134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:56.257181 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:56.258266 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.258834 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.259892 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.260104 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.260196 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:56.263230 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:56.263231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:56.263231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:56.263239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:56.264863 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:56.265479 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:56.265481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:56.265481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:56.265547 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:56.265835 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:56.265836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:56.265837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:56.265970 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:56.266053 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:56.266421 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 00:09:56.266520 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:09:56.266861 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:56.256043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33149 (local address 127.16.99.194:60341)
0812 00:09:56.256273 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:56.256319 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:56.257017 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:56.257254 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:56.257259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:56.257900 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:56.258132 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:56.258139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.258937 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.258938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:56.259792 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:56.259794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.260210 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.260211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:56.260354 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:56.261005 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:56.261016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:56.263073 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:56.265007 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:56.265018 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:56.265021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:56.265375 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:56.265619 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:56.265620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:56.265620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:56.265760 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:56.266043 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:56.266051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:56.266220 (+ 169us) client_negotiation.cc:769] Sending connection context
0812 00:09:56.266338 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:09:56.266558 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260812 00:09:56.267691 19763 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33149
I20260812 00:09:56.267817 19763 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:56.268013 19763 heartbeater.cc:511] Master 127.16.99.254:33149 requested a full tablet report, sending...
I20260812 00:09:56.268504 19409 ts_manager.cc:194] Registered new tserver with Master: 3d6df94c55634ba7b6231bbdb0392890 (127.16.99.194:41595)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
I20260812 00:09:56.268965 19409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:60341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:09:56.289611 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33149
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:56.384598 19768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:56.384871 19768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:56.384902 19768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:56.386284 19768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:56.386404 19768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:56.386461 19768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:09:56.388203 19768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:56.388456 19768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:56.388693 19768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:56.394776 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:56.394825 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:56.394992 19768 server_base.cc:1034] running on GCE node
W20260812 00:09:56.394805 19776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:56.395457 19768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:56.395808 19768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:56.397017 19768 hybrid_clock.cc:648] HybridClock initialized: now 1786493396396994 us; error 40 us; skew 500 ppm
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:56.400321 19768 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:09:56.400863 19768 webserver.cc:460] Webserver started at http://127.16.99.195:42215/ using document root <none> and password file <none>
I20260812 00:09:56.401124 19768 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:56.401198 19768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:56.401419 19768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:56.402837 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "2284a8f99b3d4770a0f1f6703f73a509"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "9c79ac91aeeb3765264286768136b522"
server_key_iv: "08fbeb937d28af26c14ff924f6393920"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.403218 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "2284a8f99b3d4770a0f1f6703f73a509"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "9c79ac91aeeb3765264286768136b522"
server_key_iv: "08fbeb937d28af26c14ff924f6393920"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.404693 19768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:09:56.405567 19784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.405846 19768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:56.406046 19768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "2284a8f99b3d4770a0f1f6703f73a509"
format_stamp: "Formatted at 2026-08-12 00:09:56 on dist-test-slave-zpfg"
server_key: "9c79ac91aeeb3765264286768136b522"
server_key_iv: "08fbeb937d28af26c14ff924f6393920"
server_key_version: "encryptionkey@0"
I20260812 00:09:56.406152 19768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:56.425938 19768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:56.429064 19768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:56.429332 19768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:56.429625 19768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:56.429988 19768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:56.430047 19768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.430089 19768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:56.430116 19768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:56.436786 19768 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:43233
I20260812 00:09:56.436872 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:43233 every 8 connection(s)
I20260812 00:09:56.437252 19768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:56.446728 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 19768
I20260812 00:09:56.446867 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:09:56.447122 16783 external_mini_cluster.cc:1428] Setting key b65386bb84c11d4f0c68ac5cab1c9f08
I20260812 00:09:56.448843 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:56.438323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:48865 (local address 127.16.99.254:33149)
0812 00:09:56.438416 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:56.438419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:56.438572 (+ 153us) server_negotiation.cc:407] Connection header received
0812 00:09:56.439128 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:56.439129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:56.439142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:56.439218 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:56.440327 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.440863 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.442133 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.442409 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.442481 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:56.445392 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:56.445394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:56.445394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:56.445402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:56.446902 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:56.447473 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:56.447474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:56.447474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:56.447578 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:56.447934 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:56.447935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:56.447935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:56.448113 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:56.448182 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:56.448636 (+ 454us) server_negotiation.cc:299] Negotiation successful
0812 00:09:56.448741 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 00:09:56.449100 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:56.438207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33149 (local address 127.16.99.195:48865)
0812 00:09:56.438423 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:56.438476 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:56.439021 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:56.439276 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:56.439280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:56.439938 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:56.440162 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:56.440167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.441002 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.441003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:56.442021 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:56.442024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:56.442521 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:56.442523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:56.442646 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:56.443712 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:56.443722 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:56.445237 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:56.447022 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:56.447032 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:56.447035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:56.447384 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:56.447663 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:56.447664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:56.447665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:56.447862 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:56.448185 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:56.448194 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:56.448462 (+ 268us) client_negotiation.cc:769] Sending connection context
0812 00:09:56.448598 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 00:09:56.448796 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260812 00:09:56.449700 19906 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33149
I20260812 00:09:56.449802 19906 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:56.450037 19906 heartbeater.cc:511] Master 127.16.99.254:33149 requested a full tablet report, sending...
I20260812 00:09:56.450485 19409 ts_manager.cc:194] Registered new tserver with Master: 2284a8f99b3d4770a0f1f6703f73a509 (127.16.99.195:43233)
I20260812 00:09:56.450887 19409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:48865
I20260812 00:09:56.459829 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:09:56.473708 19451 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:09:56.472589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.16.99.254:33149)
0812 00:09:56.472687 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:56.472690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:56.472815 (+ 125us) server_negotiation.cc:407] Connection header received
0812 00:09:56.472916 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:56.472917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:56.472928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:56.473003 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:56.473615 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52314: BlockingRecv error: recv got EOF from 127.0.0.1:52314 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:09:56.474681 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 19474
I20260812 00:09:56.481942 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 19621
I20260812 00:09:56.487524 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 19768
I20260812 00:09:56.493984 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 19375
2026-08-12T00:09:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2090 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19919](info): set up 2 sockets
Aug 12 00:09:56 dist-test-slave-zpfg krb5kdc[19919](info): commencing operation
krb5kdc: starting...
W20260812 00:09:57.581849 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:09:57 dist-test-slave-zpfg krb5kdc[19919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:09:57.730688 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33145
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:42843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:33145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:57.815587 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:57.815829 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:57.815855 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:57.817204 19933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:09:57.817257 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:57.817273 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:57.817291 19933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:09:57.817305 19933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:09:57.818917 19933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:42843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:33145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:57.819128 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:57.819320 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:57.825080 19939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:57.825057 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:57.825047 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:57.825093 19933 server_base.cc:1034] running on GCE node
I20260812 00:09:57.825603 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:57.825958 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:57.827143 19933 hybrid_clock.cc:648] HybridClock initialized: now 1786493397827103 us; error 63 us; skew 500 ppm
Aug 12 00:09:57 dist-test-slave-zpfg krb5kdc[19919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493397, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:57.830225 19933 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:09:57.830814 19933 webserver.cc:460] Webserver started at http://127.16.99.254:33579/ using document root <none> and password file <none>
I20260812 00:09:57.831084 19933 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:57.831159 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:57.831326 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:57.832541 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "e7a2fe958d1a46418a86953c3d54ea95"
format_stamp: "Formatted at 2026-08-12 00:09:57 on dist-test-slave-zpfg"
server_key: "f944ae73c2015c09b43661964cfc850d"
server_key_iv: "e7e7f6317287e0da1068ad9f2943c36f"
server_key_version: "encryptionkey@0"
I20260812 00:09:57.832942 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "e7a2fe958d1a46418a86953c3d54ea95"
format_stamp: "Formatted at 2026-08-12 00:09:57 on dist-test-slave-zpfg"
server_key: "f944ae73c2015c09b43661964cfc850d"
server_key_iv: "e7e7f6317287e0da1068ad9f2943c36f"
server_key_version: "encryptionkey@0"
I20260812 00:09:57.834430 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:09:57.835318 19949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:57.835495 19933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:57.835609 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "e7a2fe958d1a46418a86953c3d54ea95"
format_stamp: "Formatted at 2026-08-12 00:09:57 on dist-test-slave-zpfg"
server_key: "f944ae73c2015c09b43661964cfc850d"
server_key_iv: "e7e7f6317287e0da1068ad9f2943c36f"
server_key_version: "encryptionkey@0"
I20260812 00:09:57.835737 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:57.847797 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:57.850405 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:57.850584 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:57.854780 19933 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:33145
I20260812 00:09:57.854859 20006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:33145 every 8 connection(s)
I20260812 00:09:57.855394 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:09:57.856315 20007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:57.857107 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 19933
I20260812 00:09:57.857236 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:09:57.857468 16783 external_mini_cluster.cc:1428] Setting key d36e8459e82b76239e1c4bbc66d6af27
I20260812 00:09:57.859738 20007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95: Bootstrap starting.
I20260812 00:09:57.860496 20007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:57.860848 20007 log.cc:824] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:57.861721 20007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95: No bootstrap required, opened a new log
I20260812 00:09:57.864557 20007 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a2fe958d1a46418a86953c3d54ea95" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33145 } }
I20260812 00:09:57.864768 20007 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:57.864818 20007 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a2fe958d1a46418a86953c3d54ea95, State: Initialized, Role: FOLLOWER
I20260812 00:09:57.864977 20007 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a2fe958d1a46418a86953c3d54ea95" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33145 } }
I20260812 00:09:57.865080 20007 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:09:57.865114 20007 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:09:57.865173 20007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 00:09:57 dist-test-slave-zpfg krb5kdc[19919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:57.866061 20007 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a2fe958d1a46418a86953c3d54ea95" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33145 } }
I20260812 00:09:57.866202 20007 leader_election.cc:302] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a2fe958d1a46418a86953c3d54ea95; no voters:
I20260812 00:09:57.866456 20007 leader_election.cc:288] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:09:57.866526 20013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:57.866784 20007 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:09:57.866748 20013 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [term 1 LEADER]: Becoming Leader. State: Replica: e7a2fe958d1a46418a86953c3d54ea95, State: Running, Role: LEADER
I20260812 00:09:57.867337 20013 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a2fe958d1a46418a86953c3d54ea95" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33145 } }
I20260812 00:09:57.868333 20015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a2fe958d1a46418a86953c3d54ea95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a2fe958d1a46418a86953c3d54ea95" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33145 } } }
I20260812 00:09:57.868474 20014 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a2fe958d1a46418a86953c3d54ea95. Latest consensus state: current_term: 1 leader_uuid: "e7a2fe958d1a46418a86953c3d54ea95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a2fe958d1a46418a86953c3d54ea95" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33145 } } }
I20260812 00:09:57.869227 20014 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [sys.catalog]: This master's current role is: LEADER
W20260812 00:09:57.869210 20027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:09:57.869477 20027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:09:57.869225 20015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95 [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:57.869545 20031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:09:57.869791 20031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:09:57.870776 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:57.858101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.16.99.254:33145)
0812 00:09:57.858317 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:57.858327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:57.858384 (+ 57us) server_negotiation.cc:407] Connection header received
0812 00:09:57.858941 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:57.858951 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:57.859024 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:57.859189 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:57.859956 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:57.861401 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:57.862520 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:57.862988 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:57.863080 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:57.866307 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:57.866329 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:57.866329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:57.866346 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:57.868906 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:57.869399 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:57.869402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:57.869403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:57.869517 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:57.869758 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:57.869760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:57.869760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:57.870102 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:57.870274 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:57.870416 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 00:09:57.870566 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260812 00:09:57.871745 20031 catalog_manager.cc:1347] Generated new cluster ID: d24ee20bf01e4fb9ab8a8d6f4702d46e
I20260812 00:09:57.871793 20031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:09:57.871780 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:09:57.893019 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:57.888756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.16.99.254:33145)
0812 00:09:57.888825 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:57.888827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:57.889065 (+ 238us) server_negotiation.cc:407] Connection header received
0812 00:09:57.889154 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:57.889155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:57.889167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:57.889225 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:57.889741 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:57.890272 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:57.891116 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:57.891374 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:57.891459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:57.892423 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:57.892425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:57.892425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:09:57.892655 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:57.892682 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:09:57.892689 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:57.892756 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:57.892881 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 00:09:57.892915 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:09:57.906925 20031 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:09:57.907797 20031 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:09:57.916535 20031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a2fe958d1a46418a86953c3d54ea95: Generated new TSK 0
I20260812 00:09:57.916896 20031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:09:57.920503 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 19933
2026-08-12T00:09:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1425 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:09:57 dist-test-slave-zpfg krb5kdc[20042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:09:57 dist-test-slave-zpfg krb5kdc[20042](info): set up 2 sockets
Aug 12 00:09:57 dist-test-slave-zpfg krb5kdc[20042](info): commencing operation
krb5kdc: starting...
W20260812 00:09:59.014108 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:09:59.155658 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40869
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:40869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:59.250123 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:59.250464 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:59.250510 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:59.251760 20055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:09:59.251845 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:59.251870 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:59.251905 20055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:09:59.251951 20055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:09:59.253784 20055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:40869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:59.254113 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:59.254427 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:09:59.259837 20055 server_base.cc:1034] running on GCE node
W20260812 00:09:59.259793 20062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:59.259889 20061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:59.259819 20064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:59.260241 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:59.260613 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:59.261797 20055 hybrid_clock.cc:648] HybridClock initialized: now 1786493399261783 us; error 49 us; skew 500 ppm
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:59.264668 20055 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:09:59.265264 20055 webserver.cc:460] Webserver started at http://127.16.99.254:35379/ using document root <none> and password file <none>
I20260812 00:09:59.265511 20055 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:59.265583 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:59.265753 20055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:59.266999 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "0d9259d6ef9147faab9792a7a4c33413"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "d34965b1cd0002d402f0718f53902459"
server_key_iv: "afa6556ec10d8d01e8288d6d3b0ee675"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.267396 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "0d9259d6ef9147faab9792a7a4c33413"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "d34965b1cd0002d402f0718f53902459"
server_key_iv: "afa6556ec10d8d01e8288d6d3b0ee675"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.268872 20055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:09:59.269655 20071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.269852 20055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:59.269966 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "0d9259d6ef9147faab9792a7a4c33413"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "d34965b1cd0002d402f0718f53902459"
server_key_iv: "afa6556ec10d8d01e8288d6d3b0ee675"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.270045 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:59.288475 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:59.291124 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:59.291316 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:59.295773 20055 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:40869
I20260812 00:09:59.295823 20127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:40869 every 8 connection(s)
I20260812 00:09:59.296247 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:09:59.297399 20128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:09:59.299907 20128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413: Bootstrap starting.
I20260812 00:09:59.300491 20128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413: Neither blocks nor log segments found. Creating new log.
I20260812 00:09:59.300742 20128 log.cc:824] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413: Log is configured to *not* fsync() on all Append() calls
I20260812 00:09:59.301486 20128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413: No bootstrap required, opened a new log
I20260812 00:09:59.302582 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 20055
I20260812 00:09:59.302697 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:09:59.302886 16783 external_mini_cluster.cc:1428] Setting key f9634f9be72a28fe28da5ba579ba0e73
I20260812 00:09:59.303623 20128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9259d6ef9147faab9792a7a4c33413" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40869 } }
I20260812 00:09:59.303831 20128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:09:59.303869 20128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9259d6ef9147faab9792a7a4c33413, State: Initialized, Role: FOLLOWER
I20260812 00:09:59.304001 20128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9259d6ef9147faab9792a7a4c33413" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40869 } }
I20260812 00:09:59.304091 20128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:09:59.304126 20128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:09:59.304217 20128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:09:59.305022 20128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9259d6ef9147faab9792a7a4c33413" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40869 } }
I20260812 00:09:59.305133 20128 leader_election.cc:302] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d9259d6ef9147faab9792a7a4c33413; no voters:
I20260812 00:09:59.305316 20128 leader_election.cc:288] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:09:59.305389 20134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:09:59.305583 20134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [term 1 LEADER]: Becoming Leader. State: Replica: 0d9259d6ef9147faab9792a7a4c33413, State: Running, Role: LEADER
I20260812 00:09:59.305586 20128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:09:59.306411 20134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9259d6ef9147faab9792a7a4c33413" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40869 } }
I20260812 00:09:59.307085 20136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d9259d6ef9147faab9792a7a4c33413. Latest consensus state: current_term: 1 leader_uuid: "0d9259d6ef9147faab9792a7a4c33413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9259d6ef9147faab9792a7a4c33413" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40869 } } }
I20260812 00:09:59.307094 20135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d9259d6ef9147faab9792a7a4c33413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d9259d6ef9147faab9792a7a4c33413" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40869 } } }
I20260812 00:09:59.307250 20135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [sys.catalog]: This master's current role is: LEADER
I20260812 00:09:59.307229 20136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413 [sys.catalog]: This master's current role is: LEADER
W20260812 00:09:59.307987 20148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:09:59.308085 20148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:09:59.308236 20149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:09:59.308426 20149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:09:59.310308 20149 catalog_manager.cc:1347] Generated new cluster ID: 893251c296064bf3a3a08ccc3e70a779
I20260812 00:09:59.310384 20149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:59.315508 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.303471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.16.99.254:40869)
0812 00:09:59.303680 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:59.303690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:59.303726 (+ 36us) server_negotiation.cc:407] Connection header received
0812 00:09:59.303885 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:59.303892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:59.303949 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:59.304096 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:59.304790 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.306091 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.307309 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.307992 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.308448 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.311057 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:59.311062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:59.311062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:59.311075 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:59.313264 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.313666 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.313669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.313670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.313808 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.314017 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.314018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.314019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.314390 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:59.315057 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:59.315135 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 00:09:59.315289 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":119,"threads_started":1}
I20260812 00:09:59.323314 20149 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:09:59.324245 20149 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:09:59.333266 20149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d9259d6ef9147faab9792a7a4c33413: Generated new TSK 0
I20260812 00:09:59.333566 20149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:09:59.374796 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40869
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:59.486121 20155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:59.486403 20155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:59.486431 20155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:59.487728 20155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:59.487782 20155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:59.487810 20155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:09:59.489459 20155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:59.489728 20155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:59.490017 20155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:59.495790 20164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:59.495802 20162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:59.495820 20161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:59.496184 20155 server_base.cc:1034] running on GCE node
I20260812 00:09:59.496373 20155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:59.496641 20155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:59.497823 20155 hybrid_clock.cc:648] HybridClock initialized: now 1786493399497817 us; error 40 us; skew 500 ppm
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:59.500793 20155 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:09:59.501410 20155 webserver.cc:460] Webserver started at http://127.16.99.193:36397/ using document root <none> and password file <none>
I20260812 00:09:59.501662 20155 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:59.501735 20155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:59.501959 20155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:59.503152 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "4d8d6df791254184b5be2e46e2d06392"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "0de48b5be35090febc932ad5444858c0"
server_key_iv: "d0fef540418e91a5579e414a3e0b8275"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.503545 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "4d8d6df791254184b5be2e46e2d06392"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "0de48b5be35090febc932ad5444858c0"
server_key_iv: "d0fef540418e91a5579e414a3e0b8275"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.505065 20155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:09:59.505995 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.506217 20155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:59.506328 20155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "4d8d6df791254184b5be2e46e2d06392"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "0de48b5be35090febc932ad5444858c0"
server_key_iv: "d0fef540418e91a5579e414a3e0b8275"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.506428 20155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:59.527428 20155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:59.530148 20155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:59.530279 20155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:59.530637 20155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:59.531016 20155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:59.531077 20155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.531134 20155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:59.531160 20155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.537810 20155 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:36107
I20260812 00:09:59.537906 20297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:36107 every 8 connection(s)
I20260812 00:09:59.538291 20155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:09:59.542757 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 20155
I20260812 00:09:59.542863 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:09:59.543095 16783 external_mini_cluster.cc:1428] Setting key 27cea171c97abad496b900ff6e6272ea
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:59.551120 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.539220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:38973 (local address 127.16.99.254:40869)
0812 00:09:59.539291 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:59.539293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:59.539498 (+ 205us) server_negotiation.cc:407] Connection header received
0812 00:09:59.540064 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:59.540066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:59.540077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:59.540240 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:59.541243 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.541867 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.543299 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.543622 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.543721 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.546899 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:59.546901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:59.546901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:59.546911 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:59.548754 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.549531 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.549534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.549534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.549617 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.549977 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.549978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.549978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.550164 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:59.550252 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:59.550868 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 00:09:59.550990 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:09:59.551292 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.539203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40869 (local address 127.16.99.193:38973)
0812 00:09:59.539369 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:59.539409 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:59.539940 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:59.540214 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:59.540218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:59.540867 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:59.541078 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:59.541084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.542053 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.542054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:59.543199 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:59.543202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.543738 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.543739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:59.543899 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.544567 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:59.544579 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:59.546772 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:59.548980 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:59.548993 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:59.548996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:59.549425 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:59.549764 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:59.549764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:59.549765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:59.549898 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:59.550251 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:59.550261 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:59.550688 (+ 427us) client_negotiation.cc:769] Sending connection context
0812 00:09:59.550815 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 00:09:59.551014 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260812 00:09:59.551999 20298 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40869
I20260812 00:09:59.552127 20298 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:59.552368 20298 heartbeater.cc:511] Master 127.16.99.254:40869 requested a full tablet report, sending...
I20260812 00:09:59.553009 20091 ts_manager.cc:194] Registered new tserver with Master: 4d8d6df791254184b5be2e46e2d06392 (127.16.99.193:36107)
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
I20260812 00:09:59.553805 20091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:38973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:09:59.578819 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40869
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:59.666397 20303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:59.666679 20303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:59.666734 20303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:59.667995 20303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:59.668079 20303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:59.668128 20303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:09:59.669754 20303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:59.670054 20303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:59.670312 20303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:59.676155 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:59.676155 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:59.676177 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:59.676551 20303 server_base.cc:1034] running on GCE node
I20260812 00:09:59.676767 20303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:59.677004 20303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:59.678588 20303 hybrid_clock.cc:648] HybridClock initialized: now 1786493399678313 us; error 33 us; skew 500 ppm
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:59.681502 20303 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:09:59.681973 20303 webserver.cc:460] Webserver started at http://127.16.99.194:45451/ using document root <none> and password file <none>
I20260812 00:09:59.682210 20303 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:59.682256 20303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:59.682420 20303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:59.683501 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "96b930300b3540ae97f41063e4bcd992"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "7dd77729dc7f8d96a3e712f24137aef4"
server_key_iv: "b654ea06cc7509602887d156d03211b0"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.683823 20303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "96b930300b3540ae97f41063e4bcd992"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "7dd77729dc7f8d96a3e712f24137aef4"
server_key_iv: "b654ea06cc7509602887d156d03211b0"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.685353 20303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:09:59.686329 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.686578 20303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:09:59.686681 20303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "96b930300b3540ae97f41063e4bcd992"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "7dd77729dc7f8d96a3e712f24137aef4"
server_key_iv: "b654ea06cc7509602887d156d03211b0"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.686746 20303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:59.704938 20303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:59.707852 20303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:59.707983 20303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:59.708204 20303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:59.708621 20303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:59.708664 20303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.708689 20303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:59.708731 20303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.715356 20303 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:36389
I20260812 00:09:59.715519 20440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:36389 every 8 connection(s)
I20260812 00:09:59.715864 20303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:59.725517 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 20303
I20260812 00:09:59.725658 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:09:59.725901 16783 external_mini_cluster.cc:1428] Setting key 57fd5d03f655a7bc89cd38d86b1d84de
I20260812 00:09:59.727075 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.717045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:43727 (local address 127.16.99.254:40869)
0812 00:09:59.717117 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:59.717119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:59.717131 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:09:59.717663 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:59.717664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:59.717675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:59.717758 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:59.719066 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.719592 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.720743 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.720947 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.721016 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.723759 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:59.723760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:59.723761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:59.723769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:59.725125 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.725840 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.725841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.725841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.725902 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.726240 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.726241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.726241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.726436 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:59.726514 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:59.726912 (+ 398us) server_negotiation.cc:299] Negotiation successful
0812 00:09:59.727009 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:09:59.727408 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.716691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40869 (local address 127.16.99.194:43727)
0812 00:09:59.716941 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:59.716992 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:59.717562 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:59.717902 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:59.717907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:59.718700 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:59.718956 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:59.718962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.719685 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.719686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:59.720638 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:59.720640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.721047 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.721048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:59.721168 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.721844 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:59.721853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:59.723635 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:59.725327 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:59.725339 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:59.725342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:59.725740 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:59.725977 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:59.725978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:59.725979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:59.726159 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:59.726537 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:59.726544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:59.726743 (+ 199us) client_negotiation.cc:769] Sending connection context
0812 00:09:59.726885 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 00:09:59.727119 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":155,"threads_started":1}
I20260812 00:09:59.728044 20441 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40869
I20260812 00:09:59.728297 20441 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:59.728519 20441 heartbeater.cc:511] Master 127.16.99.254:40869 requested a full tablet report, sending...
I20260812 00:09:59.729005 20091 ts_manager.cc:194] Registered new tserver with Master: 96b930300b3540ae97f41063e4bcd992 (127.16.99.194:36389)
I20260812 00:09:59.729481 20091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:43727
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:09:59.760083 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40869
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:09:59.850919 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:09:59.851138 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:09:59.851166 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:09:59.852399 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:09:59.852447 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:09:59.852474 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:09:59.854080 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:09:59.854354 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:09:59.854668 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:09:59.860263 20454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:59.860394 20447 server_base.cc:1034] running on GCE node
W20260812 00:09:59.860424 20456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:09:59.860369 20453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:09:59.860886 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:09:59.861194 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:09:59.862406 20447 hybrid_clock.cc:648] HybridClock initialized: now 1786493399862371 us; error 46 us; skew 500 ppm
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:09:59.865250 20447 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:09:59.865689 20447 webserver.cc:460] Webserver started at http://127.16.99.195:35673/ using document root <none> and password file <none>
I20260812 00:09:59.865907 20447 fs_manager.cc:359] Metadata directory not provided
I20260812 00:09:59.865950 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:09:59.866045 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:09:59.867230 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "caecbfec1ac64aa29786142fdf24b7d5"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "de1c28a895be37e3b100fee200626e5b"
server_key_iv: "3666fc9fa7f02153956c5070a31ed043"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.867554 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "caecbfec1ac64aa29786142fdf24b7d5"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "de1c28a895be37e3b100fee200626e5b"
server_key_iv: "3666fc9fa7f02153956c5070a31ed043"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.868965 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:09:59.869824 20463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.870139 20447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:09:59.870239 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "caecbfec1ac64aa29786142fdf24b7d5"
format_stamp: "Formatted at 2026-08-12 00:09:59 on dist-test-slave-zpfg"
server_key: "de1c28a895be37e3b100fee200626e5b"
server_key_iv: "3666fc9fa7f02153956c5070a31ed043"
server_key_version: "encryptionkey@0"
I20260812 00:09:59.870390 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:09:59.882771 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:09:59.885699 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:09:59.885923 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:09:59.886225 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:09:59.886651 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:09:59.886714 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.886826 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:09:59.886859 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:09:59.893465 20447 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:46553
I20260812 00:09:59.893505 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:46553 every 8 connection(s)
I20260812 00:09:59.893923 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:09:59.896881 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 20447
I20260812 00:09:59.896993 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:09:59.897168 16783 external_mini_cluster.cc:1428] Setting key f4360282bf941dc99b2ad4c82a484471
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:59.905565 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.894972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:56265 (local address 127.16.99.254:40869)
0812 00:09:59.895068 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:59.895071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:59.895334 (+ 263us) server_negotiation.cc:407] Connection header received
0812 00:09:59.895836 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:59.895837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:59.895848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:59.895911 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:59.897079 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.897799 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.899140 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.899335 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.899401 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.902100 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:59.902102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:59.902103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:59.902110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:59.903700 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.904333 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.904334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.904335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.904401 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.904716 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.904717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.904717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.904845 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:59.904901 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:59.905359 (+ 458us) server_negotiation.cc:299] Negotiation successful
0812 00:09:59.905442 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 00:09:59.905728 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.894916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40869 (local address 127.16.99.195:56265)
0812 00:09:59.895092 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 00:09:59.895132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:09:59.895713 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:09:59.895991 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:09:59.896005 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:09:59.896735 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:09:59.896945 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:59.896950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.897919 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.897920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:59.899049 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:09:59.899052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.899425 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.899426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:09:59.899534 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.900121 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:09:59.900130 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:09:59.901982 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:09:59.903811 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:59.903820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:59.903822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:59.904099 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:59.904490 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:09:59.904491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:09:59.904491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:09:59.904644 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:09:59.904914 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:09:59.904921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:09:59.905075 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 00:09:59.905265 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 00:09:59.905431 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260812 00:09:59.906525 20584 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40869
I20260812 00:09:59.906659 20584 heartbeater.cc:464] Registering TS with master...
I20260812 00:09:59.906989 20584 heartbeater.cc:511] Master 127.16.99.254:40869 requested a full tablet report, sending...
I20260812 00:09:59.907691 20091 ts_manager.cc:194] Registered new tserver with Master: caecbfec1ac64aa29786142fdf24b7d5 (127.16.99.195:46553)
I20260812 00:09:59.908219 20091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:56265
I20260812 00:09:59.909260 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 00:09:59 dist-test-slave-zpfg krb5kdc[20042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493399, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:09:59.936751 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:09:59.927893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.16.99.254:40869)
0812 00:09:59.927987 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 00:09:59.927989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:09:59.928095 (+ 106us) server_negotiation.cc:407] Connection header received
0812 00:09:59.928185 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:09:59.928187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:09:59.928207 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:09:59.928283 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:09:59.929084 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.929640 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.930570 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:09:59.930796 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:09:59.930903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:09:59.933570 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:09:59.933572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:09:59.933573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:09:59.933591 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:09:59.935293 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.935760 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.935762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.935762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.935830 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:09:59.936039 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:09:59.936040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:09:59.936041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:09:59.936238 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:09:59.936365 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 00:09:59.936517 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 00:09:59.936624 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 00:09:59.937071 20091 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37038
I20260812 00:09:59.937654 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 20155
I20260812 00:09:59.944434 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 20303
I20260812 00:09:59.952042 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 20447
I20260812 00:09:59.957480 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 20055
2026-08-12T00:09:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2038 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:00 dist-test-slave-zpfg krb5kdc[20596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:00 dist-test-slave-zpfg krb5kdc[20596](info): set up 2 sockets
Aug 12 00:10:00 dist-test-slave-zpfg krb5kdc[20596](info): commencing operation
krb5kdc: starting...
W20260812 00:10:01.085733 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:10:01.226758 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42797
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:01.314400 20609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:01.314774 20609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:01.314828 20609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:01.316104 20609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:01.316179 20609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:01.316202 20609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:01.316224 20609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:01.316257 20609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:01.317921 20609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:42797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:01.318257 20609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:01.318539 20609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:01.323988 20614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:01.324010 20615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:01.324052 20609 server_base.cc:1034] running on GCE node
W20260812 00:10:01.324010 20617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:01.324551 20609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:01.324942 20609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:01.326133 20609 hybrid_clock.cc:648] HybridClock initialized: now 1786493401326122 us; error 124 us; skew 500 ppm
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:01.336512 20609 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:10:01.337208 20609 webserver.cc:460] Webserver started at http://127.16.99.254:32923/ using document root <none> and password file <none>
I20260812 00:10:01.337510 20609 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:01.337601 20609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:01.337774 20609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:01.339080 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "5bd0a51ffaf9443da39190783f693f49"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "db6c35bb88eb6bab4660d8a6c5d6c295"
server_key_iv: "38a1df35f71d8f6c693c201b8285ba82"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.339496 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "5bd0a51ffaf9443da39190783f693f49"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "db6c35bb88eb6bab4660d8a6c5d6c295"
server_key_iv: "38a1df35f71d8f6c693c201b8285ba82"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.341111 20609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:10:01.342065 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.342555 20609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:01.342720 20609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "5bd0a51ffaf9443da39190783f693f49"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "db6c35bb88eb6bab4660d8a6c5d6c295"
server_key_iv: "38a1df35f71d8f6c693c201b8285ba82"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.342820 20609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:01.365552 20609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:01.368743 20609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:01.368947 20609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:01.373898 20609 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:42797
I20260812 00:10:01.373934 20679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:42797 every 8 connection(s)
I20260812 00:10:01.374583 20609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:01.375461 20680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:01.378064 20680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49: Bootstrap starting.
I20260812 00:10:01.378834 20680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:01.379194 20680 log.cc:824] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:01.380129 20680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49: No bootstrap required, opened a new log
I20260812 00:10:01.382040 20680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd0a51ffaf9443da39190783f693f49" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42797 } }
I20260812 00:10:01.382244 20680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:01.382270 20680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd0a51ffaf9443da39190783f693f49, State: Initialized, Role: FOLLOWER
I20260812 00:10:01.382499 20680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd0a51ffaf9443da39190783f693f49" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42797 } }
I20260812 00:10:01.382567 20680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:01.382591 20680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:01.382663 20680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:01.383832 20680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd0a51ffaf9443da39190783f693f49" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42797 } }
I20260812 00:10:01.383885 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 20609
I20260812 00:10:01.383980 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:01.383981 20680 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd0a51ffaf9443da39190783f693f49; no voters:
I20260812 00:10:01.384178 16783 external_mini_cluster.cc:1428] Setting key f1461f91a2c141816c4af28ceffce8bf
I20260812 00:10:01.384192 20680 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:01.384282 20683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:01.384482 20683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd0a51ffaf9443da39190783f693f49, State: Running, Role: LEADER
I20260812 00:10:01.384516 20680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:01.384600 20683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd0a51ffaf9443da39190783f693f49" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42797 } }
I20260812 00:10:01.385046 20684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd0a51ffaf9443da39190783f693f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd0a51ffaf9443da39190783f693f49" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42797 } } }
I20260812 00:10:01.385133 20684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:01.385356 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd0a51ffaf9443da39190783f693f49. Latest consensus state: current_term: 1 leader_uuid: "5bd0a51ffaf9443da39190783f693f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd0a51ffaf9443da39190783f693f49" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42797 } } }
I20260812 00:10:01.385433 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:01.385412 20696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:01.386310 20696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:01.388207 20696 catalog_manager.cc:1347] Generated new cluster ID: 5974364af62449ae8eda0a3b7a0f86a8
I20260812 00:10:01.388281 20696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:01.392956 20696 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:01.393625 20696 catalog_manager.cc:1504] Loading token signing keys...
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:01.400202 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.384899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.16.99.254:42797)
0812 00:10:01.385102 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:01.385115 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:01.385623 (+ 508us) server_negotiation.cc:407] Connection header received
0812 00:10:01.385926 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:01.385936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:01.386012 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:01.386158 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:01.387334 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.390979 (+ 3645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.391956 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.392486 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.392600 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.395885 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:01.395893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:01.395894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:01.395911 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:01.398179 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.398641 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.398644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.398645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.398784 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.399002 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.399003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.399003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.399331 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:01.399505 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:01.399587 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:10:01.399910 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260812 00:10:01.402668 20696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd0a51ffaf9443da39190783f693f49: Generated new TSK 0
I20260812 00:10:01.402881 20696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:10:01.438458 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42797
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:01.525099 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:01.525332 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:01.525359 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:01.526662 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:01.526710 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:01.526739 20709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:01.528347 20709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:01.528618 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:01.528872 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:01.534248 20716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:01.534220 20715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:01.534220 20718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:01.534729 20709 server_base.cc:1034] running on GCE node
I20260812 00:10:01.535000 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:01.535321 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:01.536532 20709 hybrid_clock.cc:648] HybridClock initialized: now 1786493401536523 us; error 41 us; skew 500 ppm
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:01.539844 20709 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:10:01.540359 20709 webserver.cc:460] Webserver started at http://127.16.99.193:35303/ using document root <none> and password file <none>
I20260812 00:10:01.540601 20709 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:01.540652 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:01.540766 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:01.541952 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "4feb9c861f0248afac547ca0aa6acd0e"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "34c5a1c53bec4a484aa41d3075ecb9c9"
server_key_iv: "f7bbf6d8c871b87a25808dd4d68e6690"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.542317 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "4feb9c861f0248afac547ca0aa6acd0e"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "34c5a1c53bec4a484aa41d3075ecb9c9"
server_key_iv: "f7bbf6d8c871b87a25808dd4d68e6690"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.543982 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:01.544874 20726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.545075 20709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:01.545158 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "4feb9c861f0248afac547ca0aa6acd0e"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "34c5a1c53bec4a484aa41d3075ecb9c9"
server_key_iv: "f7bbf6d8c871b87a25808dd4d68e6690"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.545218 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:01.560201 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:01.562927 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:01.563062 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:01.563284 20709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:01.563656 20709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:01.563699 20709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.563724 20709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:01.563786 20709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.570473 20709 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:39983
I20260812 00:10:01.570552 20848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:39983 every 8 connection(s)
I20260812 00:10:01.570933 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:01.574965 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 20709
I20260812 00:10:01.575091 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:01.575314 16783 external_mini_cluster.cc:1428] Setting key 1eef8bef11c66062608e371a5fc693e3
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:01.582846 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.571763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:43319 (local address 127.16.99.254:42797)
0812 00:10:01.571847 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:01.571849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:01.572064 (+ 215us) server_negotiation.cc:407] Connection header received
0812 00:10:01.573142 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:01.573143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:01.573154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:01.573222 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:01.574037 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.574586 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.576340 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.576533 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.576592 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.579394 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:01.579396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:01.579396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:01.579405 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:01.581115 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.581607 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.581608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.581609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.581681 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.582012 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.582013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.582014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.582150 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:01.582256 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:01.582626 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 00:10:01.582728 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:10:01.583076 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.571713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42797 (local address 127.16.99.193:43319)
0812 00:10:01.571902 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:01.571952 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:01.573025 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:01.573409 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:01.573413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:01.573719 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:01.573945 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:01.573950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.574708 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.574710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:01.576168 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:01.576172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.576629 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.576630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:01.576802 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.577408 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:01.577420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:01.579270 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:01.581231 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:01.581247 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:01.581250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:01.581515 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:01.581777 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:01.581778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:01.581778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:01.581920 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:01.582253 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:01.582263 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:01.582472 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 00:10:01.582599 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 00:10:01.582787 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260812 00:10:01.583684 20849 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42797
I20260812 00:10:01.583798 20849 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:01.584015 20849 heartbeater.cc:511] Master 127.16.99.254:42797 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
I20260812 00:10:01.584712 20642 ts_manager.cc:194] Registered new tserver with Master: 4feb9c861f0248afac547ca0aa6acd0e (127.16.99.193:39983)
I20260812 00:10:01.585433 20642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:43319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:10:01.606751 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42797
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:01.692371 20855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:01.692633 20855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:01.692677 20855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:01.693900 20855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:01.693980 20855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:01.694024 20855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:01.695698 20855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:01.696015 20855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:01.696354 20855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:01.701715 20860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:01.701767 20861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:01.701715 20863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:01.702028 20855 server_base.cc:1034] running on GCE node
I20260812 00:10:01.702266 20855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:01.702540 20855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:01.703686 20855 hybrid_clock.cc:648] HybridClock initialized: now 1786493401703648 us; error 53 us; skew 500 ppm
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:01.706537 20855 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:10:01.707007 20855 webserver.cc:460] Webserver started at http://127.16.99.194:38063/ using document root <none> and password file <none>
I20260812 00:10:01.707226 20855 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:01.707269 20855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:01.707381 20855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:01.708447 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "00b0eef280124c79a965e80c564ea3e7"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "4c98994520950f9259e9219d837b2539"
server_key_iv: "c994e51082b16589bf60d01a80a14a65"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.708817 20855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "00b0eef280124c79a965e80c564ea3e7"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "4c98994520950f9259e9219d837b2539"
server_key_iv: "c994e51082b16589bf60d01a80a14a65"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.710472 20855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:01.711288 20871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.711588 20855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:01.711681 20855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "00b0eef280124c79a965e80c564ea3e7"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "4c98994520950f9259e9219d837b2539"
server_key_iv: "c994e51082b16589bf60d01a80a14a65"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.711762 20855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:01.724634 20855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:01.727246 20855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:01.727438 20855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:01.727730 20855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:01.728075 20855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:01.728111 20855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.728134 20855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:01.728183 20855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.734635 20855 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:42937
I20260812 00:10:01.734742 20994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:42937 every 8 connection(s)
I20260812 00:10:01.735101 20855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:01.742966 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 20855
I20260812 00:10:01.743104 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:01.743300 16783 external_mini_cluster.cc:1428] Setting key 66b2b36f0abf25b873c30bb7a9510f13
I20260812 00:10:01.746259 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.736049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:38697 (local address 127.16.99.254:42797)
0812 00:10:01.736205 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:01.736206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:01.736379 (+ 173us) server_negotiation.cc:407] Connection header received
0812 00:10:01.737402 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:01.737404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:01.737416 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:01.737496 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:01.738153 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.738850 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.739929 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.740159 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.740228 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.742795 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:01.742796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:01.742797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:01.742804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:01.744364 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.744911 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.744912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.744913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.744982 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.745475 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.745477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.745477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.745631 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:01.745701 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:01.746064 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 00:10:01.746153 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260812 00:10:01.746657 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.735998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42797 (local address 127.16.99.194:38697)
0812 00:10:01.736196 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:01.736242 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:01.737253 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:01.737556 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:01.737559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:01.737856 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:01.738061 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:01.738066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.738946 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.738946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:01.739835 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:01.739838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.740258 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.740259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:01.740410 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.741024 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:01.741032 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:01.742685 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:01.744453 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:01.744461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:01.744463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:01.744806 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:01.745223 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:01.745224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:01.745224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:01.745397 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:01.745708 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:01.745715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:01.745907 (+ 192us) client_negotiation.cc:769] Sending connection context
0812 00:10:01.746036 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 00:10:01.746261 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260812 00:10:01.747411 20995 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42797
I20260812 00:10:01.747532 20995 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:01.747742 20995 heartbeater.cc:511] Master 127.16.99.254:42797 requested a full tablet report, sending...
I20260812 00:10:01.748212 20642 ts_manager.cc:194] Registered new tserver with Master: 00b0eef280124c79a965e80c564ea3e7 (127.16.99.194:42937)
I20260812 00:10:01.748771 20642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:38697
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:10:01.775018 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42797
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:01.866968 21000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:01.867221 21000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:01.867252 21000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:01.868633 21000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:01.868690 21000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:01.868722 21000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:01.870675 21000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:01.871004 21000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:01.871291 21000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:01.877257 21005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:01.877283 21006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:01.877415 21008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:01.877518 21000 server_base.cc:1034] running on GCE node
I20260812 00:10:01.877753 21000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:01.878005 21000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:01.879189 21000 hybrid_clock.cc:648] HybridClock initialized: now 1786493401879152 us; error 52 us; skew 500 ppm
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:01.882285 21000 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:10:01.882892 21000 webserver.cc:460] Webserver started at http://127.16.99.195:33171/ using document root <none> and password file <none>
I20260812 00:10:01.883172 21000 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:01.883221 21000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:01.883405 21000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:01.884617 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "31fe707855a5484ba3cccb202157ae62"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "7e816ab983cb34fe07a2b89c6d54cf3e"
server_key_iv: "06fc54c1c8dcaa2d31c187f41b6e95a9"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.884980 21000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "31fe707855a5484ba3cccb202157ae62"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "7e816ab983cb34fe07a2b89c6d54cf3e"
server_key_iv: "06fc54c1c8dcaa2d31c187f41b6e95a9"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.886713 21000 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260812 00:10:01.887756 21016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.887984 21000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:01.888089 21000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "31fe707855a5484ba3cccb202157ae62"
format_stamp: "Formatted at 2026-08-12 00:10:01 on dist-test-slave-zpfg"
server_key: "7e816ab983cb34fe07a2b89c6d54cf3e"
server_key_iv: "06fc54c1c8dcaa2d31c187f41b6e95a9"
server_key_version: "encryptionkey@0"
I20260812 00:10:01.888273 21000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:01.912248 21000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:01.915040 21000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:01.915325 21000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:01.915601 21000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:01.916021 21000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:01.916074 21000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.916105 21000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:01.916118 21000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:01.922580 21000 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:44281
I20260812 00:10:01.922644 21134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:44281 every 8 connection(s)
I20260812 00:10:01.923002 21000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:01.931885 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21000
I20260812 00:10:01.932008 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:01.932193 16783 external_mini_cluster.cc:1428] Setting key 54ab4093a9e11ed42d8892b6477ee514
I20260812 00:10:01.934475 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.923878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:37911 (local address 127.16.99.254:42797)
0812 00:10:01.923968 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:01.923971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:01.924077 (+ 106us) server_negotiation.cc:407] Connection header received
0812 00:10:01.925065 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:01.925067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:01.925078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:01.925143 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:01.925907 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.926520 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.927572 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.927770 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.927840 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.930174 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:01.930176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:01.930176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:01.930184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:01.932405 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.933093 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.933095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.933095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.933174 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.933644 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.933645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.933645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.933772 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:01.933832 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:01.934241 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 00:10:01.934327 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 00:10:01.934784 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.923766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42797 (local address 127.16.99.195:37911)
0812 00:10:01.923938 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:01.923988 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:01.924937 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:01.925300 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:01.925305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:01.925600 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:01.925810 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:01.925816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.926615 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.926616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:01.927483 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:01.927485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.927860 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.927860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:01.927969 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.928518 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:01.928527 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:01.930063 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:01.932617 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:01.932628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:01.932630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:01.932977 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:01.933410 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:01.933411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:01.933411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:01.933580 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:01.933874 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:01.933883 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:01.934039 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 00:10:01.934165 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 00:10:01.934455 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260812 00:10:01.935382 21135 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42797
I20260812 00:10:01.935488 21135 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:01.935659 21135 heartbeater.cc:511] Master 127.16.99.254:42797 requested a full tablet report, sending...
I20260812 00:10:01.936026 20642 ts_manager.cc:194] Registered new tserver with Master: 31fe707855a5484ba3cccb202157ae62 (127.16.99.195:44281)
I20260812 00:10:01.936529 20642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:37911
I20260812 00:10:01.944792 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:10:01.971537 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.962547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.16.99.193:39983)
0812 00:10:01.962732 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:01.962738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:01.962758 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:10:01.962788 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:01.962790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:01.962804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:01.962874 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:01.963590 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.964370 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.965082 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.965441 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.965498 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.968264 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:01.968275 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:01.968275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:01.968316 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:01.969976 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.970549 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.970552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.970552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.970612 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.970832 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.970832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.970833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.970978 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:01.971093 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:01.971345 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 00:10:01.971413 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
Aug 12 00:10:01 dist-test-slave-zpfg krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:01.981061 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:01.973318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.16.99.254:42797)
0812 00:10:01.973386 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:01.973388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:01.973396 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:01.973439 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:01.973440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:01.973449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:01.973497 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:01.974143 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.974707 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.975506 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:01.975684 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:01.975735 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:01.978216 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:01.978218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:01.978218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:01.978226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:01.979836 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.980192 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.980193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.980193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.980252 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:01.980459 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:01.980460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:01.980460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:01.980618 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:01.980729 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:01.980861 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 00:10:01.980948 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 00:10:01.981380 20642 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56974
I20260812 00:10:01.981849 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 20709
I20260812 00:10:01.990149 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 20855
I20260812 00:10:01.996057 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21000
I20260812 00:10:02.003674 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 20609
2026-08-12T00:10:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2045 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:02 dist-test-slave-zpfg krb5kdc[21147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:02 dist-test-slave-zpfg krb5kdc[21147](info): set up 2 sockets
Aug 12 00:10:02 dist-test-slave-zpfg krb5kdc[21147](info): commencing operation
krb5kdc: starting...
W20260812 00:10:03.094117 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:10:03.238696 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:44959
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:44959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:03.326252 21160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:03.326586 21160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:03.326610 21160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:03.327809 21160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:03.327852 21160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:03.327865 21160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:03.327878 21160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:03.327890 21160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:03.329474 21160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:44959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:44959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:03.329704 21160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:03.329896 21160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:03.335248 21167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:03.335259 21168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:03.335268 21170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:03.335518 21160 server_base.cc:1034] running on GCE node
I20260812 00:10:03.335773 21160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:03.336161 21160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:03.337352 21160 hybrid_clock.cc:648] HybridClock initialized: now 1786493403337326 us; error 41 us; skew 500 ppm
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:03.340104 21160 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:10:03.340435 21160 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:03.340482 21160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:03.340579 21160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:03.341656 21160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "9fa3c6d169c545768723a6f4cad34596"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "b529b9a28c3266aad6337b64ff42aecb"
server_key_iv: "1c43ebd13f6980ac018bdd4ffec4ab8c"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.341981 21160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "9fa3c6d169c545768723a6f4cad34596"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "b529b9a28c3266aad6337b64ff42aecb"
server_key_iv: "1c43ebd13f6980ac018bdd4ffec4ab8c"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.343438 21160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:03.344347 21176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.344560 21160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:03.344694 21160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "9fa3c6d169c545768723a6f4cad34596"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "b529b9a28c3266aad6337b64ff42aecb"
server_key_iv: "1c43ebd13f6980ac018bdd4ffec4ab8c"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.344791 21160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:03.361052 21160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:03.364015 21160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:03.364212 21160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:03.369148 21160 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:44959
I20260812 00:10:03.369194 21230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:44959 every 8 connection(s)
I20260812 00:10:03.369546 21160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:03.370805 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:03.373450 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596: Bootstrap starting.
I20260812 00:10:03.374152 21231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:03.374534 21231 log.cc:824] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:03.375272 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21160
I20260812 00:10:03.375398 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:03.375464 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596: No bootstrap required, opened a new log
I20260812 00:10:03.375584 16783 external_mini_cluster.cc:1428] Setting key 9f039388a6184c80fc19514ed56884e1
I20260812 00:10:03.378764 21231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3c6d169c545768723a6f4cad34596" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44959 } }
I20260812 00:10:03.379014 21231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:03.379069 21231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa3c6d169c545768723a6f4cad34596, State: Initialized, Role: FOLLOWER
I20260812 00:10:03.379240 21231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3c6d169c545768723a6f4cad34596" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44959 } }
I20260812 00:10:03.379330 21231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:03.379382 21231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:03.379438 21231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:03.380319 21231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3c6d169c545768723a6f4cad34596" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44959 } }
I20260812 00:10:03.380465 21231 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa3c6d169c545768723a6f4cad34596; no voters:
I20260812 00:10:03.380710 21231 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:03.380739 21237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:03.380879 21237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa3c6d169c545768723a6f4cad34596, State: Running, Role: LEADER
I20260812 00:10:03.380972 21237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3c6d169c545768723a6f4cad34596" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44959 } }
I20260812 00:10:03.381040 21231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:03.381350 21238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa3c6d169c545768723a6f4cad34596. Latest consensus state: current_term: 1 leader_uuid: "9fa3c6d169c545768723a6f4cad34596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3c6d169c545768723a6f4cad34596" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44959 } } }
I20260812 00:10:03.381647 21238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:03.381562 21239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa3c6d169c545768723a6f4cad34596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa3c6d169c545768723a6f4cad34596" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44959 } } }
I20260812 00:10:03.382432 21239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596 [sys.catalog]: This master's current role is: LEADER
W20260812 00:10:03.383082 21253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:10:03.383143 21253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:10:03.383194 21247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:03.383397 21247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:03.386832 21247 catalog_manager.cc:1347] Generated new cluster ID: 4c9495464a2d42b1b5255cb1db4f3168
I20260812 00:10:03.386905 21247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:03.388655 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.376196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.16.99.254:44959)
0812 00:10:03.376424 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.376435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.376480 (+ 45us) server_negotiation.cc:407] Connection header received
0812 00:10:03.377053 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.377060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.377132 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.377348 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:03.378247 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.379426 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.380585 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.381284 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.381487 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.384548 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:03.384553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:03.384553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:03.384569 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:03.386812 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.387271 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.387275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.387275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.387433 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.387672 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.387672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.387673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.388011 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:03.388149 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.388277 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.388434 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":114,"threads_started":1}
I20260812 00:10:03.397585 21247 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:03.398201 21247 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:03.404778 21247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596: Generated new TSK 0
I20260812 00:10:03.405040 21247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:10:03.439581 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:44959
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:03.531665 21258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:03.531924 21258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:03.531970 21258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:03.533351 21258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:03.533432 21258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:03.533484 21258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:03.535301 21258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:03.535601 21258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:03.535903 21258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:03.541628 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:03.541646 21258 server_base.cc:1034] running on GCE node
W20260812 00:10:03.541617 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:03.541617 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:03.542212 21258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:03.542565 21258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:03.543843 21258 hybrid_clock.cc:648] HybridClock initialized: now 1786493403543830 us; error 37 us; skew 500 ppm
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:03.547067 21258 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:10:03.547544 21258 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:03.547624 21258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:03.547806 21258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:03.549073 21258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "23b8337ea4804a6c8a7967b7f884d423"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "740a424f6a23a99178c140e32f985941"
server_key_iv: "3ec0f94f1472c1ffaed7e42ff0b8b138"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.549469 21258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "23b8337ea4804a6c8a7967b7f884d423"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "740a424f6a23a99178c140e32f985941"
server_key_iv: "3ec0f94f1472c1ffaed7e42ff0b8b138"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.551085 21258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:03.552150 21273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.552520 21258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:03.552649 21258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "23b8337ea4804a6c8a7967b7f884d423"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "740a424f6a23a99178c140e32f985941"
server_key_iv: "3ec0f94f1472c1ffaed7e42ff0b8b138"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.552742 21258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:03.569568 21258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:03.572693 21258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:03.572952 21258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:03.573282 21258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:03.573685 21258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:03.573764 21258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.573815 21258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:03.573843 21258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.581439 21258 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:34849
I20260812 00:10:03.581473 21394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:34849 every 8 connection(s)
I20260812 00:10:03.581871 21258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:03.587369 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21258
I20260812 00:10:03.587471 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:03.587675 16783 external_mini_cluster.cc:1428] Setting key 5e206865400983bb52eb6ac905b2736b
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:03.593343 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.582900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:53631 (local address 127.16.99.254:44959)
0812 00:10:03.582982 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.582984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.583165 (+ 181us) server_negotiation.cc:407] Connection header received
0812 00:10:03.583660 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.583661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.583670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.583736 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:03.584982 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.585506 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.586647 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.586889 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.586953 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.589620 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:03.589622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:03.589622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:03.589631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:03.591627 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.592116 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.592118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.592118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.592191 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.592495 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.592496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.592496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.592666 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:03.592746 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.593049 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.593172 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:10:03.593426 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.582852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:44959 (local address 127.16.99.193:53631)
0812 00:10:03.583027 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:03.583072 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:03.583573 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:03.583820 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:03.583824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:03.584131 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:03.584856 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.584862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.585628 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.585629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.586551 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.586554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.586982 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.586983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.587098 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.587840 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:03.587850 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:03.589510 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:03.591723 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:03.591732 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:03.591734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:03.592028 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:03.592287 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:03.592288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:03.592288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:03.592419 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:03.592736 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:03.592741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:03.592902 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 00:10:03.592995 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:10:03.593185 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260812 00:10:03.594048 21395 heartbeater.cc:347] Connected to a master server at 127.16.99.254:44959
I20260812 00:10:03.594156 21395 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:03.594386 21395 heartbeater.cc:511] Master 127.16.99.254:44959 requested a full tablet report, sending...
I20260812 00:10:03.595044 21193 ts_manager.cc:194] Registered new tserver with Master: 23b8337ea4804a6c8a7967b7f884d423 (127.16.99.193:34849)
I20260812 00:10:03.595710 21193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:53631
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:10:03.618912 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:44959
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:03.715441 21401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:03.715755 21401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:03.715807 21401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:03.717300 21401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:03.717394 21401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:03.717450 21401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:03.719295 21401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:03.719625 21401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:03.719966 21401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:03.725878 21407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:03.726040 21409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:03.725912 21401 server_base.cc:1034] running on GCE node
W20260812 00:10:03.725878 21406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:03.726648 21401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:03.726986 21401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:03.728169 21401 hybrid_clock.cc:648] HybridClock initialized: now 1786493403728155 us; error 35 us; skew 500 ppm
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:03.731210 21401 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:10:03.731715 21401 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:03.731804 21401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:03.731980 21401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:03.733248 21401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "dd70c2f2f00b496694cf2000f1067c6c"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "fa75a25678d2a325d7a5852b0aa3f5cb"
server_key_iv: "09ed24e1c77adfff9c0f38bb62c86638"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.733677 21401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "dd70c2f2f00b496694cf2000f1067c6c"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "fa75a25678d2a325d7a5852b0aa3f5cb"
server_key_iv: "09ed24e1c77adfff9c0f38bb62c86638"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.735273 21401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:03.736289 21415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.736501 21401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:03.736618 21401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "dd70c2f2f00b496694cf2000f1067c6c"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "fa75a25678d2a325d7a5852b0aa3f5cb"
server_key_iv: "09ed24e1c77adfff9c0f38bb62c86638"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.736701 21401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:03.752254 21401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:03.755283 21401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:03.755481 21401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:03.755767 21401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:03.756244 21401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:03.756289 21401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.756316 21401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:03.756385 21401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.764887 21401 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:35703
I20260812 00:10:03.764986 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:35703 every 8 connection(s)
I20260812 00:10:03.765309 21401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:10:03.766727 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21401
I20260812 00:10:03.766853 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:03.767035 16783 external_mini_cluster.cc:1428] Setting key d05f887c52f8890ffd8faf012089dfe1
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
I20260812 00:10:03.778390 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.766180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:49193 (local address 127.16.99.254:44959)
0812 00:10:03.766253 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.766255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.766554 (+ 299us) server_negotiation.cc:407] Connection header received
0812 00:10:03.767169 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.767171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.767184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.767267 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:03.768483 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.769240 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.771115 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.771320 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.771385 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.774678 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:03.774680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:03.774680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:03.774688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:03.776348 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.776995 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.776996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.776996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.777060 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.777436 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.777437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.777438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.777661 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:03.777744 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.778165 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.778267 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:10:03.778525 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.766158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:44959 (local address 127.16.99.194:49193)
0812 00:10:03.766377 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:03.766424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:03.767009 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:03.767349 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:03.767354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:03.767769 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:03.768372 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.768378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.769400 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.769402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.770999 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.771003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.771424 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.771425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.771590 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.772191 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:03.772202 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:03.774542 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:03.776467 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:03.776480 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:03.776483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:03.776881 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:03.777137 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:03.777138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:03.777138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:03.777310 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:03.777765 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:03.777774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:03.777979 (+ 205us) client_negotiation.cc:769] Sending connection context
0812 00:10:03.778077 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 00:10:03.778231 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260812 00:10:03.779196 21541 heartbeater.cc:347] Connected to a master server at 127.16.99.254:44959
I20260812 00:10:03.779300 21541 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:03.779484 21541 heartbeater.cc:511] Master 127.16.99.254:44959 requested a full tablet report, sending...
I20260812 00:10:03.779991 21193 ts_manager.cc:194] Registered new tserver with Master: dd70c2f2f00b496694cf2000f1067c6c (127.16.99.194:35703)
I20260812 00:10:03.780427 21193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:49193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:10:03.801558 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:44959
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:03.889109 21546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:03.889384 21546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:03.889433 21546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:03.890805 21546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:03.890878 21546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:03.890926 21546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:03.892625 21546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:44959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:03.892942 21546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:03.893200 21546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:03.898659 21552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:03.898659 21551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:03.898872 21546 server_base.cc:1034] running on GCE node
W20260812 00:10:03.898723 21555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:03.899147 21546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:03.899379 21546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:03.900537 21546 hybrid_clock.cc:648] HybridClock initialized: now 1786493403900524 us; error 29 us; skew 500 ppm
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:03.903589 21546 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:10:03.904058 21546 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:03.904114 21546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:03.904259 21546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:03.905553 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "d565b4a6e96b434fa7e73803fd8a95f6"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "96fa35b2c6bc91f1bf89c6e77b0b9a36"
server_key_iv: "6b5d5216997a6e952fe37fb4af250b10"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.905946 21546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "d565b4a6e96b434fa7e73803fd8a95f6"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "96fa35b2c6bc91f1bf89c6e77b0b9a36"
server_key_iv: "6b5d5216997a6e952fe37fb4af250b10"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.907632 21546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:03.908694 21561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.908999 21546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:03.909103 21546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "d565b4a6e96b434fa7e73803fd8a95f6"
format_stamp: "Formatted at 2026-08-12 00:10:03 on dist-test-slave-zpfg"
server_key: "96fa35b2c6bc91f1bf89c6e77b0b9a36"
server_key_iv: "6b5d5216997a6e952fe37fb4af250b10"
server_key_version: "encryptionkey@0"
I20260812 00:10:03.909214 21546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:03.930085 21546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:03.933386 21546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:03.933566 21546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:03.933876 21546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:03.934247 21546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:03.934310 21546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.934378 21546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:03.934428 21546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:03.942576 21546 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:35661
I20260812 00:10:03.942678 21682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:35661 every 8 connection(s)
I20260812 00:10:03.943001 21546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:03.948446 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21546
I20260812 00:10:03.948590 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:03.948832 16783 external_mini_cluster.cc:1428] Setting key bcd01f98ec96bbdb95a3eccd5121b01c
Aug 12 00:10:03 dist-test-slave-zpfg krb5kdc[21147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:03.954546 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.943898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:50025 (local address 127.16.99.254:44959)
0812 00:10:03.943968 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.943970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.944177 (+ 207us) server_negotiation.cc:407] Connection header received
0812 00:10:03.944800 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.944802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.944815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.944887 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:03.946113 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.946639 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.947823 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.948400 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.948511 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.951482 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:03.951484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:03.951484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:03.951491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:03.952886 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.953431 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.953432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.953432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.953493 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.953752 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.953753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.953753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.953872 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:03.953926 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.954327 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.954475 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:10:03.954852 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.943813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:44959 (local address 127.16.99.195:50025)
0812 00:10:03.944009 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:03.944060 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:03.944676 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:03.944976 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:03.944980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:03.945298 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:03.945979 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.945987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.946739 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.946740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.947725 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.947728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.948521 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.948522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.948710 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.949378 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:03.949388 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:03.951350 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:03.952993 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:03.953001 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:03.953003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:03.953354 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:03.953554 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:03.953555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:03.953555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:03.953690 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:03.953958 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:03.953966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:03.954106 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 00:10:03.954207 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 00:10:03.954525 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":111,"threads_started":1}
I20260812 00:10:03.955646 21683 heartbeater.cc:347] Connected to a master server at 127.16.99.254:44959
I20260812 00:10:03.955766 21683 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:03.955948 21683 heartbeater.cc:511] Master 127.16.99.254:44959 requested a full tablet report, sending...
I20260812 00:10:03.956406 21193 ts_manager.cc:194] Registered new tserver with Master: d565b4a6e96b434fa7e73803fd8a95f6 (127.16.99.195:35661)
I20260812 00:10:03.956911 21193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:50025
I20260812 00:10:03.961481 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:10:03.970679 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.963000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.16.99.254:44959)
0812 00:10:03.963047 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.963053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.963120 (+ 67us) server_negotiation.cc:407] Connection header received
0812 00:10:03.963253 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.963254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.963263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.963328 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:03.964088 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.964765 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.965902 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.966381 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.966464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.968008 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:03.968009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:03.968009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:03.968017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:03.969501 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.969876 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.969877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.969877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.969938 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:03.970115 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:03.970115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:03.970115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:03.970272 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:03.970434 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.970532 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.970605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:10:03.972321 21193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:03.973999 21193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:03.981681 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.977573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:35703 (local address 127.0.0.1:48382)
0812 00:10:03.977793 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:03.977804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:03.977848 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:03.978363 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:03.978365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:03.978367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:03.978420 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.978421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.979484 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.979485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.980418 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.980421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.981322 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.981323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.981413 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.981431 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 00:10:03.981515 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:10:03.981571 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":10,"thread_start_us":100,"threads_started":1}
I20260812 00:10:03.981698 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.977684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.16.99.194:35703)
0812 00:10:03.978037 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.978042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.978053 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:03.978092 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.978094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.978250 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.978361 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:03.978511 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.979378 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.980506 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.981235 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.981312 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.981373 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.981544 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.981593 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":117,"threads_started":1}
I20260812 00:10:03.982227 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.976957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34849 (local address 127.0.0.1:57594)
0812 00:10:03.977097 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:03.977140 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:03.977209 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:03.977546 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:03.977550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:03.977553 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:03.977647 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.977650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.979062 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.979063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.980651 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.980654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.981674 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.981675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.981757 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.981762 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:10:03.982057 (+ 295us) client_negotiation.cc:240] Negotiation successful
0812 00:10:03.982116 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":92,"threads_started":1}
I20260812 00:10:03.982784 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.977141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.16.99.193:34849)
0812 00:10:03.977328 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.977334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.977347 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:03.977393 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.977395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.977412 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.977504 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:03.977747 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.978945 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.980748 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.981562 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.982532 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.982611 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.982649 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.982698 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260812 00:10:03.982744 21464 tablet_service.cc:1467] Processing CreateTablet for tablet bd1a5247800a4f3db60d7ea4b79d7404 (DEFAULT_TABLE table=test-table [id=caad8737dde9462f81afc90675c5d0bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:03.982918 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.977846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.16.99.195:35661)
0812 00:10:03.978097 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:03.978102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:03.978114 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:03.978156 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:03.978158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:03.978191 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:03.978300 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:03.978567 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.979628 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.981034 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.982179 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.982471 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.982542 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:03.982758 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 00:10:03.982811 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
I20260812 00:10:03.983204 21464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1a5247800a4f3db60d7ea4b79d7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:03.983228 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:03.977272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:35661 (local address 127.0.0.1:40672)
0812 00:10:03.977407 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:03.977419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:03.977476 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:03.978426 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:03.978426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:03.978429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:03.978504 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.978506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.979779 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.979780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.980933 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:03.980935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:03.982574 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:03.982575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:03.982648 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:03.982653 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:10:03.982714 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 00:10:03.982763 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260812 00:10:03.983942 21613 tablet_service.cc:1467] Processing CreateTablet for tablet bd1a5247800a4f3db60d7ea4b79d7404 (DEFAULT_TABLE table=test-table [id=caad8737dde9462f81afc90675c5d0bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:03.984252 21613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1a5247800a4f3db60d7ea4b79d7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:03.986820 21324 tablet_service.cc:1467] Processing CreateTablet for tablet bd1a5247800a4f3db60d7ea4b79d7404 (DEFAULT_TABLE table=test-table [id=caad8737dde9462f81afc90675c5d0bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:03.987128 21701 tablet_bootstrap.cc:492] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Bootstrap starting.
I20260812 00:10:03.987195 21324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1a5247800a4f3db60d7ea4b79d7404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:03.987943 21701 tablet_bootstrap.cc:654] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:03.988687 21701 log.cc:824] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:03.989362 21703 tablet_bootstrap.cc:492] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Bootstrap starting.
I20260812 00:10:03.990100 21701 tablet_bootstrap.cc:492] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: No bootstrap required, opened a new log
I20260812 00:10:03.990147 21703 tablet_bootstrap.cc:654] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:03.990192 21701 ts_tablet_manager.cc:1397] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:10:03.990840 21703 log.cc:824] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:03.991101 21704 tablet_bootstrap.cc:492] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: Bootstrap starting.
I20260812 00:10:03.991824 21704 tablet_bootstrap.cc:654] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:03.991905 21703 tablet_bootstrap.cc:492] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: No bootstrap required, opened a new log
I20260812 00:10:03.991984 21703 ts_tablet_manager.cc:1397] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 00:10:03.992441 21704 log.cc:824] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:03.992558 21701 raft_consensus.cc:348] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:03.992769 21701 raft_consensus.cc:374] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:03.992810 21701 raft_consensus.cc:729] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d565b4a6e96b434fa7e73803fd8a95f6, State: Initialized, Role: FOLLOWER
I20260812 00:10:03.992951 21701 consensus_queue.cc:260] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:03.993294 21683 heartbeater.cc:503] Master 127.16.99.254:44959 was elected leader, sending a full tablet report...
I20260812 00:10:03.993546 21701 ts_tablet_manager.cc:1428] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:10:03.993978 21704 tablet_bootstrap.cc:492] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: No bootstrap required, opened a new log
I20260812 00:10:03.994078 21704 ts_tablet_manager.cc:1397] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:10:03.994500 21703 raft_consensus.cc:348] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:03.994700 21703 raft_consensus.cc:374] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:03.994757 21703 raft_consensus.cc:729] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd70c2f2f00b496694cf2000f1067c6c, State: Initialized, Role: FOLLOWER
I20260812 00:10:03.994916 21703 consensus_queue.cc:260] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:03.995287 21703 ts_tablet_manager.cc:1428] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260812 00:10:03.995317 21541 heartbeater.cc:503] Master 127.16.99.254:44959 was elected leader, sending a full tablet report...
I20260812 00:10:03.996537 21704 raft_consensus.cc:348] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:03.996711 21704 raft_consensus.cc:374] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:03.996762 21704 raft_consensus.cc:729] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b8337ea4804a6c8a7967b7f884d423, State: Initialized, Role: FOLLOWER
I20260812 00:10:03.996893 21704 consensus_queue.cc:260] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:03.997313 21704 ts_tablet_manager.cc:1428] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:10:03.997334 21395 heartbeater.cc:503] Master 127.16.99.254:44959 was elected leader, sending a full tablet report...
W20260812 00:10:04.015861 21542 tablet.cc:2369] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:10:04.082871 21396 tablet.cc:2369] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:04.153164 21708 raft_consensus.cc:482] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:04.153331 21708 raft_consensus.cc:504] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:04.153772 21708 leader_election.cc:288] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b8337ea4804a6c8a7967b7f884d423 (127.16.99.193:34849), d565b4a6e96b434fa7e73803fd8a95f6 (127.16.99.195:35661)
I20260812 00:10:04.157527 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.154152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:41539 (local address 127.16.99.193:34849)
0812 00:10:04.154217 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.154220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.154355 (+ 135us) server_negotiation.cc:407] Connection header received
0812 00:10:04.154461 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.154463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.154474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.154537 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:04.154726 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.155387 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.156681 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.157090 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.157164 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:04.157175 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.157388 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.157446 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:10:04.157562 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.154264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:52293 (local address 127.16.99.195:35661)
0812 00:10:04.154324 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.154327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.154482 (+ 155us) server_negotiation.cc:407] Connection header received
0812 00:10:04.154576 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.154578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.154589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.154724 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:04.154887 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.155404 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.156577 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.157083 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.157193 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:04.157202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.157436 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.157491 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 00:10:04.157548 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.154016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34849 (local address 127.16.99.194:41539)
0812 00:10:04.154230 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.154245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.154373 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.154556 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.154557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.154559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:04.154645 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:04.154648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.155492 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.155494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:04.156571 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:04.156574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.157194 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.157196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:04.157301 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:04.157305 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.157354 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.157413 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 00:10:04.157562 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.154173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:35661 (local address 127.16.99.194:52293)
0812 00:10:04.154393 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.154405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.154475 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.154726 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.154727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.154729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:04.154810 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:04.154812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.155525 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.155526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:04.156486 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:04.156489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.157256 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.157256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:04.157327 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:04.157331 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.157413 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.157453 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260812 00:10:04.158034 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1a5247800a4f3db60d7ea4b79d7404" candidate_uuid: "dd70c2f2f00b496694cf2000f1067c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" is_pre_election: true
I20260812 00:10:04.158156 21345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1a5247800a4f3db60d7ea4b79d7404" candidate_uuid: "dd70c2f2f00b496694cf2000f1067c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8337ea4804a6c8a7967b7f884d423" is_pre_election: true
I20260812 00:10:04.158231 21633 raft_consensus.cc:2454] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70c2f2f00b496694cf2000f1067c6c in term 0.
I20260812 00:10:04.158397 21345 raft_consensus.cc:2454] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd70c2f2f00b496694cf2000f1067c6c in term 0.
I20260812 00:10:04.158491 21416 leader_election.cc:302] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d565b4a6e96b434fa7e73803fd8a95f6, dd70c2f2f00b496694cf2000f1067c6c; no voters:
I20260812 00:10:04.158663 21708 raft_consensus.cc:2781] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:04.158774 21708 raft_consensus.cc:482] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:04.158815 21708 raft_consensus.cc:3037] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:04.159605 21708 raft_consensus.cc:504] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:04.159782 21708 leader_election.cc:288] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [CANDIDATE]: Term 1 election: Requested vote from peers 23b8337ea4804a6c8a7967b7f884d423 (127.16.99.193:34849), d565b4a6e96b434fa7e73803fd8a95f6 (127.16.99.195:35661)
I20260812 00:10:04.160038 21345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1a5247800a4f3db60d7ea4b79d7404" candidate_uuid: "dd70c2f2f00b496694cf2000f1067c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8337ea4804a6c8a7967b7f884d423"
I20260812 00:10:04.160104 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1a5247800a4f3db60d7ea4b79d7404" candidate_uuid: "dd70c2f2f00b496694cf2000f1067c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d565b4a6e96b434fa7e73803fd8a95f6"
I20260812 00:10:04.160169 21345 raft_consensus.cc:3037] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:04.160238 21633 raft_consensus.cc:3037] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:04.161077 21345 raft_consensus.cc:2454] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70c2f2f00b496694cf2000f1067c6c in term 1.
I20260812 00:10:04.161264 21416 leader_election.cc:302] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b8337ea4804a6c8a7967b7f884d423, dd70c2f2f00b496694cf2000f1067c6c; no voters:
I20260812 00:10:04.161386 21708 raft_consensus.cc:2781] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:04.161482 21633 raft_consensus.cc:2454] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd70c2f2f00b496694cf2000f1067c6c in term 1.
I20260812 00:10:04.161585 21708 raft_consensus.cc:686] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 1 LEADER]: Becoming Leader. State: Replica: dd70c2f2f00b496694cf2000f1067c6c, State: Running, Role: LEADER
I20260812 00:10:04.161782 21708 consensus_queue.cc:237] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } }
I20260812 00:10:04.162976 21193 catalog_manager.cc:5495] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c reported cstate change: term changed from 0 to 1, leader changed from <none> to dd70c2f2f00b496694cf2000f1067c6c (127.16.99.194). New cstate: current_term: 1 leader_uuid: "dd70c2f2f00b496694cf2000f1067c6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd70c2f2f00b496694cf2000f1067c6c" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35703 } health_report { overall_health: HEALTHY } } }
W20260812 00:10:04.193709 21684 tablet.cc:2369] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:04.211480 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.208317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.16.99.194:35703)
0812 00:10:04.208388 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.208390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.208399 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:04.208483 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.208485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.208515 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.208730 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:04.208907 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.209588 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.210502 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:04.210748 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:04.210840 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:04.211039 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:04.211166 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:04.211278 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.211299 (+ 21us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.211324 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:10:04.213735 21633 raft_consensus.cc:1261] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 1 FOLLOWER]: Refusing update from remote peer dd70c2f2f00b496694cf2000f1067c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:04.213831 21345 raft_consensus.cc:1261] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 1 FOLLOWER]: Refusing update from remote peer dd70c2f2f00b496694cf2000f1067c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:04.214255 21711 consensus_queue.cc:1035] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d565b4a6e96b434fa7e73803fd8a95f6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 35661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:04.214480 21708 consensus_queue.cc:1035] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b8337ea4804a6c8a7967b7f884d423" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:04.217415 21716 mvcc.cc:205] Tried to move back new op lower bound from 7317476983656243200 to 7317476983447552000. Current Snapshot: MvccSnapshot[applied={T|T < 7317476983656243200}]
I20260812 00:10:04.217975 21715 mvcc.cc:205] Tried to move back new op lower bound from 7317476983656243200 to 7317476983447552000. Current Snapshot: MvccSnapshot[applied={T|T < 7317476983656243200}]
I20260812 00:10:04.220878 21193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:04.221005 21193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:04.222113 21193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fa3c6d169c545768723a6f4cad34596: Sending DeleteTablet for 3 replicas of tablet bd1a5247800a4f3db60d7ea4b79d7404
I20260812 00:10:04.222599 21324 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1a5247800a4f3db60d7ea4b79d7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:04 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:57594
I20260812 00:10:04.222893 21613 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1a5247800a4f3db60d7ea4b79d7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:04 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:40672
I20260812 00:10:04.223133 21464 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1a5247800a4f3db60d7ea4b79d7404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:04 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:48382
I20260812 00:10:04.223174 21724 tablet_replica.cc:330] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423: stopping tablet replica
I20260812 00:10:04.223325 21724 raft_consensus.cc:2229] T bd1a5247800a4f3db60d7ea4b79d7404 P 23b8337ea4804a6c8a7967b7f884d423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:04.223384 21726 tablet_replica.cc:330] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: stopping tablet replica
I20260812 00:10:04.223394 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21258
I20260812 00:10:04.223531 21726 raft_consensus.cc:2229] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:04.223703 21726 raft_consensus.cc:2258] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:04.224884 21726 ts_tablet_manager.cc:1905] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:04.227221 21726 ts_tablet_manager.cc:1918] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:04.227393 21726 log.cc:1196] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/bd1a5247800a4f3db60d7ea4b79d7404
I20260812 00:10:04.227658 21725 tablet_replica.cc:330] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: stopping tablet replica
I20260812 00:10:04.227813 21726 ts_tablet_manager.cc:1939] T bd1a5247800a4f3db60d7ea4b79d7404 P dd70c2f2f00b496694cf2000f1067c6c: Deleting consensus metadata
I20260812 00:10:04.227864 21725 raft_consensus.cc:2229] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:04.228073 21725 raft_consensus.cc:2258] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:04.228559 21177 catalog_manager.cc:4841] TS dd70c2f2f00b496694cf2000f1067c6c (127.16.99.194:35703): tablet bd1a5247800a4f3db60d7ea4b79d7404 (table test-table [id=caad8737dde9462f81afc90675c5d0bb]) successfully deleted
I20260812 00:10:04.228691 21725 ts_tablet_manager.cc:1905] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:04.231078 21725 ts_tablet_manager.cc:1918] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:04.231197 21725 log.cc:1196] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/bd1a5247800a4f3db60d7ea4b79d7404
I20260812 00:10:04.231597 21725 ts_tablet_manager.cc:1939] T bd1a5247800a4f3db60d7ea4b79d7404 P d565b4a6e96b434fa7e73803fd8a95f6: Deleting consensus metadata
I20260812 00:10:04.232621 21177 catalog_manager.cc:4841] TS d565b4a6e96b434fa7e73803fd8a95f6 (127.16.99.195:35661): tablet bd1a5247800a4f3db60d7ea4b79d7404 (table test-table [id=caad8737dde9462f81afc90675c5d0bb]) successfully deleted
W20260812 00:10:04.233763 21177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:34849 (error 108)
I20260812 00:10:04.234478 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.234282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34849 (local address 127.0.0.1:57608)
0812 00:10:04.234380 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.234431 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:34849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 00:10:04.234577 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21401
W20260812 00:10:04.237586 21177 catalog_manager.cc:4584] TS 23b8337ea4804a6c8a7967b7f884d423 (127.16.99.193:34849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd1a5247800a4f3db60d7ea4b79d7404: Network error: Client connection negotiation failed: client connection to 127.16.99.193:34849: connect: Connection refused (error 111)
I20260812 00:10:04.241513 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21546
I20260812 00:10:04.248494 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21160
2026-08-12T00:10:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2245 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T00:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:04Z Disabled control of system clock
I20260812 00:10:04.282536 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45417
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:38787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:45417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:10:04.373219 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:04.373483 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:04.373509 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:04.374826 21731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:04.374881 21731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:04.374894 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:04.374907 21731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:04.374920 21731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:04.376471 21731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:38787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:45417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:04.376734 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:04.377019 21731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:04.382714 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:04.382745 21731 server_base.cc:1034] running on GCE node
W20260812 00:10:04.382709 21738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:04.382705 21740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:04.383281 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:04.383641 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:04.384840 21731 hybrid_clock.cc:648] HybridClock initialized: now 1786493404384815 us; error 44 us; skew 500 ppm
I20260812 00:10:04.386507 21731 webserver.cc:460] Webserver started at http://127.16.99.254:38423/ using document root <none> and password file <none>
I20260812 00:10:04.386821 21731 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:04.386909 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:04.387087 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:04.388516 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "2e313966677944df8f83a65e7569c7ab"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "4928085971773c7ee61eaa18fa368385"
server_key_iv: "a5fde9a834d54e6b203baa261d24dff3"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.388952 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "2e313966677944df8f83a65e7569c7ab"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "4928085971773c7ee61eaa18fa368385"
server_key_iv: "a5fde9a834d54e6b203baa261d24dff3"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.390700 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 00:10:04.391932 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.392289 21731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:04.392405 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "2e313966677944df8f83a65e7569c7ab"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "4928085971773c7ee61eaa18fa368385"
server_key_iv: "a5fde9a834d54e6b203baa261d24dff3"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.392503 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:04.396654 21731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:04.397290 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:04.397487 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:04.402299 21731 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:45417
I20260812 00:10:04.402402 21799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:45417 every 8 connection(s)
I20260812 00:10:04.402822 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:04.403646 21800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:04.406916 21800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab: Bootstrap starting.
I20260812 00:10:04.407570 21800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:04.407905 21800 log.cc:824] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:04.408565 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21731
I20260812 00:10:04.408787 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:04.408987 21800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab: No bootstrap required, opened a new log
I20260812 00:10:04.409014 16783 external_mini_cluster.cc:1428] Setting key 630222735b5d1654cc348032d01ca9af
I20260812 00:10:04.411746 21800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e313966677944df8f83a65e7569c7ab" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45417 } }
I20260812 00:10:04.412016 21800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:04.412106 21800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e313966677944df8f83a65e7569c7ab, State: Initialized, Role: FOLLOWER
I20260812 00:10:04.412341 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.409660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.16.99.254:45417)
0812 00:10:04.409887 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.409895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.409922 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:10:04.410665 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.410674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.410721 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.410853 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.411651 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.411656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.411656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.411662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.411712 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.411723 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.411790 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.411896 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.411980 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260812 00:10:04.412454 21800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e313966677944df8f83a65e7569c7ab" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45417 } }
I20260812 00:10:04.412567 21800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:04.412600 21800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:04.412650 21800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:04.413597 21800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e313966677944df8f83a65e7569c7ab" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45417 } }
I20260812 00:10:04.413725 21800 leader_election.cc:302] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e313966677944df8f83a65e7569c7ab; no voters:
I20260812 00:10:04.413903 21800 leader_election.cc:288] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:04.413936 21806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:04.414149 21806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [term 1 LEADER]: Becoming Leader. State: Replica: 2e313966677944df8f83a65e7569c7ab, State: Running, Role: LEADER
I20260812 00:10:04.414222 21800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:04.414304 21806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e313966677944df8f83a65e7569c7ab" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45417 } }
I20260812 00:10:04.414808 21808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e313966677944df8f83a65e7569c7ab. Latest consensus state: current_term: 1 leader_uuid: "2e313966677944df8f83a65e7569c7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e313966677944df8f83a65e7569c7ab" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45417 } } }
I20260812 00:10:04.414812 21807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e313966677944df8f83a65e7569c7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e313966677944df8f83a65e7569c7ab" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45417 } } }
I20260812 00:10:04.415045 21808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:04.415051 21807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:04.415548 21815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:04.416093 21815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:04.417843 21815 catalog_manager.cc:1347] Generated new cluster ID: f03db8a24c924cdda417070650620433
I20260812 00:10:04.417925 21815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:04.436378 21815 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:04.436604 21815 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:04.442305 21815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab: Generated new TSK 0
I20260812 00:10:04.442551 21815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:10:04.451226 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45417
--builtin_ntp_servers=127.16.99.212:38787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:10:04.555409 21828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:04.555634 21828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:04.555660 21828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:04.556954 21828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:04.557008 21828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:04.557037 21828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:04.558668 21828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:38787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:04.558965 21828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:04.559322 21828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:04.566362 21836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:04.566450 21834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:04.566550 21833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:04.566586 21828 server_base.cc:1034] running on GCE node
I20260812 00:10:04.566833 21828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:04.567126 21828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:04.568372 21828 hybrid_clock.cc:648] HybridClock initialized: now 1786493404568346 us; error 40 us; skew 500 ppm
I20260812 00:10:04.569758 21828 webserver.cc:460] Webserver started at http://127.16.99.193:44517/ using document root <none> and password file <none>
I20260812 00:10:04.570050 21828 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:04.570129 21828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:04.570297 21828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:04.571527 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "049fdfaf57e445448be444d92981f66d"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "207d3daf05049e997498a23decc5897c"
server_key_iv: "1197c61101a915fb8de35aa2141a472a"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.571931 21828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "049fdfaf57e445448be444d92981f66d"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "207d3daf05049e997498a23decc5897c"
server_key_iv: "1197c61101a915fb8de35aa2141a472a"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.573504 21828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:04.574486 21842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.574690 21828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:04.574802 21828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "049fdfaf57e445448be444d92981f66d"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "207d3daf05049e997498a23decc5897c"
server_key_iv: "1197c61101a915fb8de35aa2141a472a"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.574891 21828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:04.578572 21828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:04.578884 21828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:04.579032 21828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:04.579245 21828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:04.579644 21828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:04.579703 21828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.579746 21828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:04.579772 21828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.586117 21828 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:40585
I20260812 00:10:04.586161 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:40585 every 8 connection(s)
I20260812 00:10:04.586617 21828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:04.588014 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21828
I20260812 00:10:04.588102 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:04.588299 16783 external_mini_cluster.cc:1428] Setting key 0a5717852f2eb4b35eb28817c6efa356
I20260812 00:10:04.589427 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45417
--builtin_ntp_servers=127.16.99.212:38787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 00:10:04.590183 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.587512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:43073 (local address 127.16.99.254:45417)
0812 00:10:04.587564 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.587571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.587856 (+ 285us) server_negotiation.cc:407] Connection header received
0812 00:10:04.588441 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.588443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.588450 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.588507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.589911 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.589913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.589913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.589918 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.589935 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.589939 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.589996 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.590095 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.590112 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:10:04.590492 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.587465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45417 (local address 127.16.99.193:43073)
0812 00:10:04.587671 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.587715 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.588318 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.588589 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.588594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.588603 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:04.589747 (+ 1144us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:04.589747 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:04.589780 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:04.589811 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:04.589993 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:04.589999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:04.590003 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.590095 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.590165 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260812 00:10:04.590965 21964 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45417
I20260812 00:10:04.591069 21964 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:04.591241 21964 heartbeater.cc:511] Master 127.16.99.254:45417 requested a full tablet report, sending...
I20260812 00:10:04.591861 21763 ts_manager.cc:194] Registered new tserver with Master: 049fdfaf57e445448be444d92981f66d (127.16.99.193:40585)
W20260812 00:10:04.675184 21967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:04.675415 21967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:04.675442 21967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:04.676767 21967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:04.676822 21967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:04.676852 21967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:04.678524 21967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:38787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:04.678829 21967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:04.679188 21967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:04.685190 21976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:04.685221 21974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:04.685439 21967 server_base.cc:1034] running on GCE node
W20260812 00:10:04.685258 21973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:04.685765 21967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:04.686065 21967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:04.687240 21967 hybrid_clock.cc:648] HybridClock initialized: now 1786493404687214 us; error 41 us; skew 500 ppm
I20260812 00:10:04.688856 21967 webserver.cc:460] Webserver started at http://127.16.99.194:43889/ using document root <none> and password file <none>
I20260812 00:10:04.689146 21967 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:04.689227 21967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:04.689399 21967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:04.690681 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "093d144d70074ea5a07e49536fea9632"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "408eff53b1b3e34a1b455d3620a3f529"
server_key_iv: "a9ca61e53afa209f054f70b9c41e4f1e"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.691121 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "093d144d70074ea5a07e49536fea9632"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "408eff53b1b3e34a1b455d3620a3f529"
server_key_iv: "a9ca61e53afa209f054f70b9c41e4f1e"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.692723 21967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:10:04.693727 21983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.693980 21967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:04.694108 21967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "093d144d70074ea5a07e49536fea9632"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "408eff53b1b3e34a1b455d3620a3f529"
server_key_iv: "a9ca61e53afa209f054f70b9c41e4f1e"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.694267 21967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:04.698167 21967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:04.698534 21967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:04.698688 21967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:04.698915 21967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:04.699283 21967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:04.699347 21967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.699389 21967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:04.699415 21967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.706446 21967 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:45847
I20260812 00:10:04.706497 22102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:45847 every 8 connection(s)
I20260812 00:10:04.706888 21967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:10:04.710803 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.707799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:55323 (local address 127.16.99.254:45417)
0812 00:10:04.707920 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.707922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.708266 (+ 344us) server_negotiation.cc:407] Connection header received
0812 00:10:04.708921 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.708922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.708929 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.708977 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.710468 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.710470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.710470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.710477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.710495 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.710500 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.710584 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.710698 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.710717 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 00:10:04.711031 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.707761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45417 (local address 127.16.99.194:55323)
0812 00:10:04.708015 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.708055 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.708783 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.709081 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.709085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.709095 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:04.710210 (+ 1115us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:04.710210 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:04.710244 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:04.710278 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:04.710570 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:04.710578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:04.710581 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.710710 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.710783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20260812 00:10:04.711627 22103 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45417
I20260812 00:10:04.711750 22103 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:04.711932 22103 heartbeater.cc:511] Master 127.16.99.254:45417 requested a full tablet report, sending...
I20260812 00:10:04.712406 21763 ts_manager.cc:194] Registered new tserver with Master: 093d144d70074ea5a07e49536fea9632 (127.16.99.194:45847)
I20260812 00:10:04.715809 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 21967
I20260812 00:10:04.715898 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:04.716048 16783 external_mini_cluster.cc:1428] Setting key 6aa4d5799b99c960316f771c0a89df03
I20260812 00:10:04.717110 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45417
--builtin_ntp_servers=127.16.99.212:38787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 00:10:04.816895 22106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:04.817196 22106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:04.817245 22106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:04.818734 22106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:04.818814 22106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:04.818867 22106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:04.820752 22106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:38787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:04.821089 22106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:04.821471 22106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:04.827510 22114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:04.827524 22111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:04.827520 22112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:04.827801 22106 server_base.cc:1034] running on GCE node
I20260812 00:10:04.828135 22106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:04.828425 22106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:04.829607 22106 hybrid_clock.cc:648] HybridClock initialized: now 1786493404829591 us; error 32 us; skew 500 ppm
I20260812 00:10:04.831182 22106 webserver.cc:460] Webserver started at http://127.16.99.195:38987/ using document root <none> and password file <none>
I20260812 00:10:04.831478 22106 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:04.831557 22106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:04.831722 22106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:04.832938 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "e8f1cc85fd3d4d1893bae0563ee65e56"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "4a1ebf4787660bf875ac99672ac9f179"
server_key_iv: "f04fb4070050e20c48f623be83d71ce4"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.833351 22106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f1cc85fd3d4d1893bae0563ee65e56"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "4a1ebf4787660bf875ac99672ac9f179"
server_key_iv: "f04fb4070050e20c48f623be83d71ce4"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.835014 22106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 00:10:04.836207 22120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.836457 22106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:04.836585 22106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "e8f1cc85fd3d4d1893bae0563ee65e56"
format_stamp: "Formatted at 2026-08-12 00:10:04 on dist-test-slave-zpfg"
server_key: "4a1ebf4787660bf875ac99672ac9f179"
server_key_iv: "f04fb4070050e20c48f623be83d71ce4"
server_key_version: "encryptionkey@0"
I20260812 00:10:04.836694 22106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:04.841126 22106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:04.841457 22106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:04.841645 22106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:04.841886 22106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:04.842254 22106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:04.842327 22106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.842422 22106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:04.842468 22106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:04.849366 22106 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:33257
I20260812 00:10:04.849443 22241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:33257 every 8 connection(s)
I20260812 00:10:04.849860 22106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:04.853430 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22106
I20260812 00:10:04.853577 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:04.853703 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.850902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:43985 (local address 127.16.99.254:45417)
0812 00:10:04.851058 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.851061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.851410 (+ 349us) server_negotiation.cc:407] Connection header received
0812 00:10:04.851964 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.851965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.851973 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.852031 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.853376 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.853378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.853378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.853383 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.853399 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.853404 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.853464 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.853599 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.853617 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 00:10:04.853787 16783 external_mini_cluster.cc:1428] Setting key 6034956dad4c21d25f86b34d00e3db53
I20260812 00:10:04.853919 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.850804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45417 (local address 127.16.99.195:43985)
0812 00:10:04.851185 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.851294 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.851848 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.852113 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.852116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.852124 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:04.853148 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:04.853149 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:04.853201 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:04.853253 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:04.853485 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:04.853492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:04.853495 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.853592 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.853667 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260812 00:10:04.854722 22242 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45417
I20260812 00:10:04.854852 22242 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:04.855033 22242 heartbeater.cc:511] Master 127.16.99.254:45417 requested a full tablet report, sending...
I20260812 00:10:04.855675 21762 ts_manager.cc:194] Registered new tserver with Master: e8f1cc85fd3d4d1893bae0563ee65e56 (127.16.99.195:33257)
I20260812 00:10:04.866418 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:10:04.869321 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.867930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.16.99.254:45417)
0812 00:10:04.868014 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.868017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.868170 (+ 153us) server_negotiation.cc:407] Connection header received
0812 00:10:04.868198 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.868199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.868205 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.868264 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.868998 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.869000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.869000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.869005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.869023 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.869027 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.869108 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.869190 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.869204 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:10:04.871445 21762 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:04.873235 21762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:04.878360 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.875828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.16.99.195:33257)
0812 00:10:04.876430 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.876436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.876447 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:04.876478 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.876480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.876491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.876567 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.877962 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.877972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.877972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.877977 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.878014 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.878045 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.878101 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.878224 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.878241 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":87,"threads_started":1}
I20260812 00:10:04.878391 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.875775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:40585 (local address 127.0.0.1:36226)
0812 00:10:04.876212 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.876223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.876294 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.876784 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.876793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.876801 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:04.877483 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:04.877484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:04.877504 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:04.877542 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:04.877807 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:04.877815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:04.877819 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.877905 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.878222 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":118,"threads_started":1}
I20260812 00:10:04.878531 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.875725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:33257 (local address 127.0.0.1:53082)
0812 00:10:04.875985 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.876037 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.876127 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.876924 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.876926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.876930 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:04.877846 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:04.877847 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:04.877856 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:04.877871 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:04.878125 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:04.878126 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:04.878127 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.878199 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.878216 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
I20260812 00:10:04.878438 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.875899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.16.99.193:40585)
0812 00:10:04.876113 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.876118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.876310 (+ 192us) server_negotiation.cc:407] Connection header received
0812 00:10:04.876350 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.876353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.876366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.876449 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.877643 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.877651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.877651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.877656 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.877698 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.877726 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.877792 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.878275 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.878295 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260812 00:10:04.879150 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.875969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:45847 (local address 127.0.0.1:57722)
0812 00:10:04.876238 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:04.876249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:04.876325 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:04.877756 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:04.877757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:04.877781 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:04.878270 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:04.878270 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:04.878277 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:04.878291 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:04.878975 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:04.878976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:04.878976 (+ 0us) client_negotiation.cc:769] Sending connection context
0812 00:10:04.879067 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 00:10:04.879081 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":72,"threads_started":1}
I20260812 00:10:04.879172 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:04.876189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.16.99.194:45847)
0812 00:10:04.877613 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:04.877617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:04.877627 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:04.877652 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:04.877654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:04.877663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:04.877751 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:04.878831 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:04.878842 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:04.878842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:04.878849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:04.878886 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:04.878914 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:04.878981 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:04.879078 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 00:10:04.879093 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":1245,"threads_started":1}
I20260812 00:10:04.879474 21895 tablet_service.cc:1467] Processing CreateTablet for tablet 09d8b676d4724ed4a03ef24ae8141a46 (DEFAULT_TABLE table=test-table [id=d6296a464f9a48f19e7ed23d549e0057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:04.879474 22171 tablet_service.cc:1467] Processing CreateTablet for tablet 09d8b676d4724ed4a03ef24ae8141a46 (DEFAULT_TABLE table=test-table [id=d6296a464f9a48f19e7ed23d549e0057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:04.879905 22171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d8b676d4724ed4a03ef24ae8141a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:04.879905 21895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d8b676d4724ed4a03ef24ae8141a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:04.880033 22033 tablet_service.cc:1467] Processing CreateTablet for tablet 09d8b676d4724ed4a03ef24ae8141a46 (DEFAULT_TABLE table=test-table [id=d6296a464f9a48f19e7ed23d549e0057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:04.880321 22033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d8b676d4724ed4a03ef24ae8141a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:04.883208 22256 tablet_bootstrap.cc:492] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: Bootstrap starting.
I20260812 00:10:04.883718 22257 tablet_bootstrap.cc:492] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: Bootstrap starting.
I20260812 00:10:04.884009 22256 tablet_bootstrap.cc:654] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:04.884366 22258 tablet_bootstrap.cc:492] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Bootstrap starting.
I20260812 00:10:04.884382 22256 log.cc:824] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:04.884430 22257 tablet_bootstrap.cc:654] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:04.884745 22257 log.cc:824] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:04.885149 22258 tablet_bootstrap.cc:654] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:04.885335 22256 tablet_bootstrap.cc:492] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: No bootstrap required, opened a new log
I20260812 00:10:04.885455 22256 ts_tablet_manager.cc:1397] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:04.885607 22257 tablet_bootstrap.cc:492] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: No bootstrap required, opened a new log
I20260812 00:10:04.885737 22257 ts_tablet_manager.cc:1397] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:04.885721 22258 log.cc:824] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:04.886605 22258 tablet_bootstrap.cc:492] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: No bootstrap required, opened a new log
I20260812 00:10:04.886749 22258 ts_tablet_manager.cc:1397] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:04.888552 22256 raft_consensus.cc:348] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:04.888635 22257 raft_consensus.cc:348] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:04.888775 22257 raft_consensus.cc:374] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:04.888764 22256 raft_consensus.cc:374] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:04.888811 22257 raft_consensus.cc:729] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049fdfaf57e445448be444d92981f66d, State: Initialized, Role: FOLLOWER
I20260812 00:10:04.888818 22256 raft_consensus.cc:729] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093d144d70074ea5a07e49536fea9632, State: Initialized, Role: FOLLOWER
I20260812 00:10:04.888957 22256 consensus_queue.cc:260] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:04.889004 22257 consensus_queue.cc:260] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:04.889155 22258 raft_consensus.cc:348] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:04.889343 22257 ts_tablet_manager.cc:1428] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:10:04.889343 22256 ts_tablet_manager.cc:1428] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 00:10:04.889495 22103 heartbeater.cc:503] Master 127.16.99.254:45417 was elected leader, sending a full tablet report...
I20260812 00:10:04.889521 21964 heartbeater.cc:503] Master 127.16.99.254:45417 was elected leader, sending a full tablet report...
I20260812 00:10:04.889400 22258 raft_consensus.cc:374] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:04.889797 22258 raft_consensus.cc:729] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f1cc85fd3d4d1893bae0563ee65e56, State: Initialized, Role: FOLLOWER
I20260812 00:10:04.889958 22258 consensus_queue.cc:260] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:04.890270 22258 ts_tablet_manager.cc:1428] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:10:04.890307 22242 heartbeater.cc:503] Master 127.16.99.254:45417 was elected leader, sending a full tablet report...
W20260812 00:10:04.957553 22104 tablet.cc:2369] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:05.074894 22264 raft_consensus.cc:482] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:05.075044 22264 raft_consensus.cc:504] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:05.075413 22264 leader_election.cc:288] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049fdfaf57e445448be444d92981f66d (127.16.99.193:40585), 093d144d70074ea5a07e49536fea9632 (127.16.99.194:45847)
I20260812 00:10:05.077535 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.075676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:40585 (local address 127.16.99.195:40341)
0812 00:10:05.075768 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:05.075785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:05.075849 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:05.076155 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:05.076157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:05.076160 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:05.076958 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:05.076959 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:05.076967 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:05.076981 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:05.077328 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:05.077329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:05.077330 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:10:05.077419 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 00:10:05.077435 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 00:10:05.077522 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.075861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:54897 (local address 127.16.99.194:45847)
0812 00:10:05.075911 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.075913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.075971 (+ 58us) server_negotiation.cc:407] Connection header received
0812 00:10:05.075997 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.075998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.076003 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.076054 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:05.077193 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:05.077195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:05.077196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:05.077202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:05.077221 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:05.077227 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:05.077315 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:05.077417 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 00:10:05.077435 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:10:05.077529 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.075773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40341 (local address 127.16.99.193:40585)
0812 00:10:05.075835 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.075838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.075853 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:10:05.075980 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.075982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.075989 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.076126 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:05.077070 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:05.077072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:05.077072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:05.077092 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:05.077112 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:05.077117 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:05.077314 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:05.077432 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 00:10:05.077448 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:10:05.077554 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.075698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:45847 (local address 127.16.99.195:54897)
0812 00:10:05.075884 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:05.075894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:05.075937 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:05.076114 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:05.076115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:05.076119 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:05.077041 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:05.077043 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:05.077052 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:05.077067 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:05.077335 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:05.077336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:05.077337 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 00:10:05.077410 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 00:10:05.077428 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260812 00:10:05.077968 21915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d8b676d4724ed4a03ef24ae8141a46" candidate_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049fdfaf57e445448be444d92981f66d" is_pre_election: true
I20260812 00:10:05.078012 22054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d8b676d4724ed4a03ef24ae8141a46" candidate_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093d144d70074ea5a07e49536fea9632" is_pre_election: true
I20260812 00:10:05.078167 21915 raft_consensus.cc:2454] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f1cc85fd3d4d1893bae0563ee65e56 in term 0.
I20260812 00:10:05.078186 22054 raft_consensus.cc:2454] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f1cc85fd3d4d1893bae0563ee65e56 in term 0.
I20260812 00:10:05.078469 22122 leader_election.cc:302] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049fdfaf57e445448be444d92981f66d, e8f1cc85fd3d4d1893bae0563ee65e56; no voters:
I20260812 00:10:05.078620 22264 raft_consensus.cc:2781] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:05.078691 22264 raft_consensus.cc:482] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:05.078776 22264 raft_consensus.cc:3037] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:05.079638 22264 raft_consensus.cc:504] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:05.079816 22264 leader_election.cc:288] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [CANDIDATE]: Term 1 election: Requested vote from peers 049fdfaf57e445448be444d92981f66d (127.16.99.193:40585), 093d144d70074ea5a07e49536fea9632 (127.16.99.194:45847)
I20260812 00:10:05.079988 21915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d8b676d4724ed4a03ef24ae8141a46" candidate_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049fdfaf57e445448be444d92981f66d"
I20260812 00:10:05.080001 22054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d8b676d4724ed4a03ef24ae8141a46" candidate_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093d144d70074ea5a07e49536fea9632"
I20260812 00:10:05.080080 21915 raft_consensus.cc:3037] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:05.080116 22054 raft_consensus.cc:3037] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:05.080931 21915 raft_consensus.cc:2454] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f1cc85fd3d4d1893bae0563ee65e56 in term 1.
I20260812 00:10:05.081066 22054 raft_consensus.cc:2454] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f1cc85fd3d4d1893bae0563ee65e56 in term 1.
I20260812 00:10:05.081136 22122 leader_election.cc:302] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049fdfaf57e445448be444d92981f66d, e8f1cc85fd3d4d1893bae0563ee65e56; no voters:
I20260812 00:10:05.081254 22264 raft_consensus.cc:2781] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:05.081444 22264 raft_consensus.cc:686] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 1 LEADER]: Becoming Leader. State: Replica: e8f1cc85fd3d4d1893bae0563ee65e56, State: Running, Role: LEADER
I20260812 00:10:05.081617 22264 consensus_queue.cc:237] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } }
I20260812 00:10:05.082746 21762 catalog_manager.cc:5495] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f1cc85fd3d4d1893bae0563ee65e56 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f1cc85fd3d4d1893bae0563ee65e56" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 } health_report { overall_health: UNKNOWN } } }
W20260812 00:10:05.087477 21965 tablet.cc:2369] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:10:05.100589 22243 tablet.cc:2369] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:05.109726 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.108599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.16.99.195:33257)
0812 00:10:05.108664 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.108666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.108678 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:05.108765 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.108766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.108781 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.108851 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:05.109404 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:05.109405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:05.109405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:05.109409 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:05.109423 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:05.109426 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:05.109472 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:05.109602 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 00:10:05.109614 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:10:05.111778 22054 raft_consensus.cc:1261] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 1 FOLLOWER]: Refusing update from remote peer e8f1cc85fd3d4d1893bae0563ee65e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:05.111817 21915 raft_consensus.cc:1261] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 1 FOLLOWER]: Refusing update from remote peer e8f1cc85fd3d4d1893bae0563ee65e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:05.112258 22266 consensus_queue.cc:1035] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093d144d70074ea5a07e49536fea9632" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:05.112455 22264 consensus_queue.cc:1035] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049fdfaf57e445448be444d92981f66d" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:05.115058 22270 mvcc.cc:205] Tried to move back new op lower bound from 7317476987334930432 to 7317476987215208448. Current Snapshot: MvccSnapshot[applied={T|T < 7317476987334930432}]
I20260812 00:10:05.115621 22272 mvcc.cc:205] Tried to move back new op lower bound from 7317476987334930432 to 7317476987215208448. Current Snapshot: MvccSnapshot[applied={T|T < 7317476987334930432}]
I20260812 00:10:05.115885 22271 mvcc.cc:205] Tried to move back new op lower bound from 7317476987334930432 to 7317476987215208448. Current Snapshot: MvccSnapshot[applied={T|T < 7317476987334930432}]
I20260812 00:10:05.118454 21761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:05.118558 21761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:05.119547 21761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e313966677944df8f83a65e7569c7ab: Sending DeleteTablet for 3 replicas of tablet 09d8b676d4724ed4a03ef24ae8141a46
I20260812 00:10:05.119946 21895 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d8b676d4724ed4a03ef24ae8141a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:05 UTC) from {username='slave'} at 127.0.0.1:36226
I20260812 00:10:05.119959 22033 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d8b676d4724ed4a03ef24ae8141a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:05 UTC) from {username='slave'} at 127.0.0.1:57722
I20260812 00:10:05.120332 22280 tablet_replica.cc:330] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d: stopping tablet replica
I20260812 00:10:05.120508 22280 raft_consensus.cc:2229] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:05.120545 22171 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d8b676d4724ed4a03ef24ae8141a46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:05 UTC) from {username='slave'} at 127.0.0.1:53082
I20260812 00:10:05.120671 22280 raft_consensus.cc:2258] T 09d8b676d4724ed4a03ef24ae8141a46 P 049fdfaf57e445448be444d92981f66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:05.120803 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21828
I20260812 00:10:05.120842 22282 tablet_replica.cc:330] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: stopping tablet replica
I20260812 00:10:05.120934 22282 raft_consensus.cc:2229] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:05.121042 22282 raft_consensus.cc:2258] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:05.121675 22282 ts_tablet_manager.cc:1905] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:05.123703 22282 ts_tablet_manager.cc:1918] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:05.123847 22282 log.cc:1196] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/09d8b676d4724ed4a03ef24ae8141a46
I20260812 00:10:05.124191 22282 ts_tablet_manager.cc:1939] T 09d8b676d4724ed4a03ef24ae8141a46 P e8f1cc85fd3d4d1893bae0563ee65e56: Deleting consensus metadata
I20260812 00:10:05.124714 21747 catalog_manager.cc:4841] TS e8f1cc85fd3d4d1893bae0563ee65e56 (127.16.99.195:33257): tablet 09d8b676d4724ed4a03ef24ae8141a46 (table test-table [id=d6296a464f9a48f19e7ed23d549e0057]) successfully deleted
I20260812 00:10:05.130924 22281 tablet_replica.cc:330] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: stopping tablet replica
I20260812 00:10:05.131152 22281 raft_consensus.cc:2229] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 00:10:05.131242 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:40585 (error 108)
I20260812 00:10:05.131361 22281 raft_consensus.cc:2258] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:05.132052 22281 ts_tablet_manager.cc:1905] T 09d8b676d4724ed4a03ef24ae8141a46 P 093d144d70074ea5a07e49536fea9632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:05.132099 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.131761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:40585 (local address 127.0.0.1:36236)
0812 00:10:05.131981 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:05.132056 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:40585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260812 00:10:05.132246 21748 catalog_manager.cc:4584] TS 049fdfaf57e445448be444d92981f66d (127.16.99.193:40585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09d8b676d4724ed4a03ef24ae8141a46: Network error: Client connection negotiation failed: client connection to 127.16.99.193:40585: connect: Connection refused (error 111)
I20260812 00:10:05.132313 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21967
I20260812 00:10:05.139876 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.139663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:45847 (local address 127.0.0.1:57724)
0812 00:10:05.139723 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:05.139754 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.194:45847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:10:05.140084 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22106
I20260812 00:10:05.147065 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 21731
2026-08-12T00:10:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (897 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 00:10:05.170478 16783 mini_oidc.cc:152] Starting JWKS server
I20260812 00:10:05.171672 16783 webserver.cc:460] Webserver started at https://127.0.0.1:44451/ using document root <none> and password file <none>
I20260812 00:10:05.171811 16783 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:10:05.172014 16783 webserver.cc:460] Webserver started at http://127.0.0.1:34463/ using document root <none> and password file <none>
2026-08-12T00:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:05Z Disabled control of system clock
I20260812 00:10:05.181955 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:36701
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:45935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:36701
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44451/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/testchainca.cert with env {}
W20260812 00:10:05.270471 22289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:05.270722 22289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:05.270748 22289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:05.271957 22289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:05.271997 22289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:05.272009 22289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:05.272024 22289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:05.272037 22289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:10:05.272049 22289 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:10:05.272061 22289 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44451/jwks.json
I20260812 00:10:05.273718 22289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:36701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:36701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44451/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:05.273972 22289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:05.274148 22289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:05.279553 22298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:05.279618 22295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:05.279867 22296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:05.280018 22289 server_base.cc:1034] running on GCE node
I20260812 00:10:05.280313 22289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:05.280654 22289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:05.281826 22289 hybrid_clock.cc:648] HybridClock initialized: now 1786493405281770 us; error 67 us; skew 500 ppm
I20260812 00:10:05.283586 22289 webserver.cc:460] Webserver started at http://127.16.99.254:44733/ using document root <none> and password file <none>
I20260812 00:10:05.283872 22289 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:05.283926 22289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:05.284057 22289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:05.285275 22289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "ab802ca696164fe7b80209573b8efc65"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "66d8b2f70db982878021a69aae39ca37"
server_key_iv: "dd11acb5bf2120cc14f7b3d130a3b6bc"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.285699 22289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "ab802ca696164fe7b80209573b8efc65"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "66d8b2f70db982878021a69aae39ca37"
server_key_iv: "dd11acb5bf2120cc14f7b3d130a3b6bc"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.287516 22289 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 00:10:05.288705 22304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:05.289079 22289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:05.289286 22289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "ab802ca696164fe7b80209573b8efc65"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "66d8b2f70db982878021a69aae39ca37"
server_key_iv: "dd11acb5bf2120cc14f7b3d130a3b6bc"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.289438 22289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:05.320722 22289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:05.321115 22289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:05.321292 22289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:05.325932 22289 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:36701
I20260812 00:10:05.325970 22358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:36701 every 8 connection(s)
I20260812 00:10:05.326407 22289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:05.327490 22359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:05.328697 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22289
I20260812 00:10:05.328786 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:05.328969 16783 external_mini_cluster.cc:1428] Setting key 4cf298dd2793a8adaa0b8cb08413e01d
I20260812 00:10:05.330497 22359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65: Bootstrap starting.
I20260812 00:10:05.331151 22359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:05.331475 22359 log.cc:824] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:05.332216 22359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65: No bootstrap required, opened a new log
I20260812 00:10:05.334230 22359 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab802ca696164fe7b80209573b8efc65" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36701 } }
I20260812 00:10:05.334475 22359 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:05.334534 22359 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab802ca696164fe7b80209573b8efc65, State: Initialized, Role: FOLLOWER
I20260812 00:10:05.334698 22359 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab802ca696164fe7b80209573b8efc65" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36701 } }
I20260812 00:10:05.334801 22359 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:05.334862 22359 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:05.334919 22359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:05.335218 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.329554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38356 (local address 127.16.99.254:36701)
0812 00:10:05.329770 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.329778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.329804 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:10:05.330397 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.330407 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.330488 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.330642 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:05.330874 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.331897 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.332824 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.333266 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.333415 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.334457 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:05.334462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:05.334462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:05.334716 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:05.334752 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:05.334764 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:05.334837 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:05.334913 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 00:10:05.335000 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260812 00:10:05.335615 22359 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab802ca696164fe7b80209573b8efc65" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36701 } }
I20260812 00:10:05.335739 22359 leader_election.cc:302] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab802ca696164fe7b80209573b8efc65; no voters:
I20260812 00:10:05.335915 22359 leader_election.cc:288] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:05.335949 22365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:05.336117 22365 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [term 1 LEADER]: Becoming Leader. State: Replica: ab802ca696164fe7b80209573b8efc65, State: Running, Role: LEADER
I20260812 00:10:05.336170 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:10:05.336221 22359 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:05.336264 22365 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab802ca696164fe7b80209573b8efc65" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36701 } }
I20260812 00:10:05.336606 22365 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab802ca696164fe7b80209573b8efc65. Latest consensus state: current_term: 1 leader_uuid: "ab802ca696164fe7b80209573b8efc65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab802ca696164fe7b80209573b8efc65" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36701 } } }
I20260812 00:10:05.336680 22365 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:05.336625 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab802ca696164fe7b80209573b8efc65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab802ca696164fe7b80209573b8efc65" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36701 } } }
I20260812 00:10:05.336750 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65 [sys.catalog]: This master's current role is: LEADER
W20260812 00:10:05.338119 22380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:10:05.338195 22380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:10:05.338258 22371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:05.338480 22371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:05.340095 22371 catalog_manager.cc:1347] Generated new cluster ID: fbe861909dc44bce9e2526696d7aee5a
I20260812 00:10:05.340154 22371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:05.350535 22371 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:05.351291 22371 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:05.359846 22371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab802ca696164fe7b80209573b8efc65: Generated new TSK 0
I20260812 00:10:05.360095 22371 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 00:10:05.399586 22391 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 00:10:05.399864 22391 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 00:10:05.399930 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.374616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.16.99.254:36701)
0812 00:10:05.374674 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.374677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.374922 (+ 245us) server_negotiation.cc:407] Connection header received
0812 00:10:05.375084 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.375085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.375098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.375185 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:10:05.375386 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.375922 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.377111 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.377347 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.377413 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.377763 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:10:05.399591 (+ 21828us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44451/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 00:10:05.399818 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38366: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44451/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260812 00:10:05.400373 22390 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:05.374505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:36701 (local address 127.0.0.1:38366)
0812 00:10:05.374790 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:05.374803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:05.374890 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:05.375205 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:05.375206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:05.375208 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:10:05.375290 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:05.375293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.376005 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.376006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:05.376997 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:05.377001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.377451 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.377452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:05.377523 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.377579 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:10:05.399775 (+ 22196us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44451/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 00:10:05.399920 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.99.254:36701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44451/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":184,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.99.254:36701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44451/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 00:10:05.400835 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22289
2026-08-12T00:10:05Z chronyd exiting
I20260812 00:10:05.439270 16783 test_util.cc:175] -----------------------------------------------
I20260812 00:10:05.439399 16783 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786493381608766-16783-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (270 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 00:10:05.440658 16783 mini_oidc.cc:152] Starting JWKS server
I20260812 00:10:05.441807 16783 webserver.cc:460] Webserver started at https://127.0.0.1:36581/ using document root <none> and password file <none>
I20260812 00:10:05.441877 16783 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:10:05.442060 16783 webserver.cc:460] Webserver started at http://127.0.0.1:44251/ using document root <none> and password file <none>
2026-08-12T00:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:05Z Disabled control of system clock
I20260812 00:10:05.451988 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:34831
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:34831
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36581/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/testchainca.cert with env {}
W20260812 00:10:05.541697 22397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:05.541980 22397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:05.542032 22397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:05.543592 22397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:05.543685 22397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:05.543715 22397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:05.543751 22397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:05.543783 22397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:10:05.543814 22397 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:10:05.543843 22397 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36581/jwks.json
I20260812 00:10:05.545629 22397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:34831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36581/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:05.545984 22397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:05.546249 22397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:10:05.552160 22397 server_base.cc:1034] running on GCE node
W20260812 00:10:05.552095 22406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:05.552084 22403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:05.552202 22404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:05.552664 22397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:05.553030 22397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:05.554214 22397 hybrid_clock.cc:648] HybridClock initialized: now 1786493405554185 us; error 46 us; skew 500 ppm
I20260812 00:10:05.555740 22397 webserver.cc:460] Webserver started at http://127.16.99.254:43347/ using document root <none> and password file <none>
I20260812 00:10:05.556048 22397 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:05.556137 22397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:05.556300 22397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:05.557516 22397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "73f1fb7fa3be93bb86b4e7d67b8f9f9d"
server_key_iv: "8fba892693b931f1bcf474ece14f02d6"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.557941 22397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "73f1fb7fa3be93bb86b4e7d67b8f9f9d"
server_key_iv: "8fba892693b931f1bcf474ece14f02d6"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.559409 22397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:05.560233 22412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:05.560416 22397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:10:05.560526 22397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "73f1fb7fa3be93bb86b4e7d67b8f9f9d"
server_key_iv: "8fba892693b931f1bcf474ece14f02d6"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.560606 22397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:05.575588 22397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:05.575973 22397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:05.576159 22397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:05.580682 22397 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:34831
I20260812 00:10:05.580722 22467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:34831 every 8 connection(s)
I20260812 00:10:05.581128 22397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:05.581899 22468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:05.585103 22468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5: Bootstrap starting.
I20260812 00:10:05.585752 22468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:05.586068 22468 log.cc:824] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:05.586942 22468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5: No bootstrap required, opened a new log
I20260812 00:10:05.587385 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22397
I20260812 00:10:05.587508 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:05.587692 16783 external_mini_cluster.cc:1428] Setting key 59dbd1558994b991ac9ecdfc51a5b5b7
I20260812 00:10:05.589151 22468 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34831 } }
I20260812 00:10:05.589332 22468 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:05.589375 22468 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb13c84b3d144e1b85f6a9c171fd5e5, State: Initialized, Role: FOLLOWER
I20260812 00:10:05.589584 22468 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34831 } }
I20260812 00:10:05.589704 22468 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:05.589748 22468 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:05.589812 22468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:05.590510 22468 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34831 } }
I20260812 00:10:05.590629 22468 leader_election.cc:302] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcb13c84b3d144e1b85f6a9c171fd5e5; no voters:
I20260812 00:10:05.590831 22468 leader_election.cc:288] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:05.590884 22473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:05.591051 22473 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [term 1 LEADER]: Becoming Leader. State: Replica: dcb13c84b3d144e1b85f6a9c171fd5e5, State: Running, Role: LEADER
I20260812 00:10:05.591120 22468 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:05.591194 22473 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34831 } }
I20260812 00:10:05.591619 22473 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcb13c84b3d144e1b85f6a9c171fd5e5. Latest consensus state: current_term: 1 leader_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34831 } } }
I20260812 00:10:05.591708 22473 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:05.591971 22483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:05.592583 22483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:05.592743 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb13c84b3d144e1b85f6a9c171fd5e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34831 } } }
I20260812 00:10:05.593588 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:05.594794 22483 catalog_manager.cc:1347] Generated new cluster ID: b1cbc11f0229499a91b6e6b506877f1d
I20260812 00:10:05.594898 22483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:05.595253 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.588275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.16.99.254:34831)
0812 00:10:05.588488 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.588496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.588521 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:10:05.589125 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.589132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.589206 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.589354 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:05.589620 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.591433 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.592792 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.593456 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.593585 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.594485 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:05.594489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:05.594489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:05.594709 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:05.594760 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:05.594775 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:05.594862 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:05.594952 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 00:10:05.595040 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260812 00:10:05.596252 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:10:05.603461 22483 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:05.604266 22483 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:05.611424 22483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcb13c84b3d144e1b85f6a9c171fd5e5: Generated new TSK 0
I20260812 00:10:05.611683 22483 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 00:10:05.635861 22500 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 00:10:05.635974 22500 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 00:10:05.636354 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.620803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.16.99.254:34831)
0812 00:10:05.620857 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.620860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.620907 (+ 47us) server_negotiation.cc:407] Connection header received
0812 00:10:05.620979 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.620980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.620994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.621085 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:10:05.621207 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.622020 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.622963 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.623274 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.623355 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.623532 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:10:05.636088 (+ 12556us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 00:10:05.636275 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 00:10:05.636426 22472 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 00:10:05.636647 22499 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:05.620707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:34831 (local address 127.0.0.1:50018)
0812 00:10:05.620843 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:05.620854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:05.620919 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:05.621087 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:05.621088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:05.621090 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:10:05.621146 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:05.621148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.622122 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.622123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:05.622861 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:05.622864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.623368 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.623369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:05.623411 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.623416 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:10:05.636353 (+ 12937us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 00:10:05.636548 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.99.254:34831: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36581/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260812 00:10:05.637006 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22397
2026-08-12T00:10:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 00:10:05.677167 16783 mini_oidc.cc:152] Starting JWKS server
I20260812 00:10:05.677917 16783 webserver.cc:460] Webserver started at https://127.0.0.1:39017/ using document root <none> and password file <none>
I20260812 00:10:05.677994 16783 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 00:10:05.678167 16783 webserver.cc:460] Webserver started at http://127.0.0.1:44053/ using document root <none> and password file <none>
2026-08-12T00:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:05Z Disabled control of system clock
I20260812 00:10:05.687929 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37871
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:37871
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39017/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 00:10:05.783921 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:05.784228 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:05.784279 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:05.785699 22507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:05.785784 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:05.785807 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:05.785831 22507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:05.785866 22507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:10:05.785893 22507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:10:05.785924 22507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39017/jwks.json
I20260812 00:10:05.787637 22507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:37871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39017/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:05.788010 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:05.788395 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:05.794108 22513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:05.794109 22516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:05.794159 22512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:05.794320 22507 server_base.cc:1034] running on GCE node
I20260812 00:10:05.794663 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:05.795043 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:05.796195 22507 hybrid_clock.cc:648] HybridClock initialized: now 1786493405796172 us; error 40 us; skew 500 ppm
I20260812 00:10:05.797606 22507 webserver.cc:460] Webserver started at http://127.16.99.254:39095/ using document root <none> and password file <none>
I20260812 00:10:05.797856 22507 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:05.797904 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:05.798003 22507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:05.799196 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "e2152b08362b4ab3b2ad5639b09ca61c"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "61c474da24f38ab6be28ed4d5e1c3c4b"
server_key_iv: "65d25386e87670db500f3dac93a8d26b"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.799531 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "e2152b08362b4ab3b2ad5639b09ca61c"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "61c474da24f38ab6be28ed4d5e1c3c4b"
server_key_iv: "65d25386e87670db500f3dac93a8d26b"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.800940 22507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:10:05.801844 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:05.802235 22507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:05.802418 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "e2152b08362b4ab3b2ad5639b09ca61c"
format_stamp: "Formatted at 2026-08-12 00:10:05 on dist-test-slave-zpfg"
server_key: "61c474da24f38ab6be28ed4d5e1c3c4b"
server_key_iv: "65d25386e87670db500f3dac93a8d26b"
server_key_version: "encryptionkey@0"
I20260812 00:10:05.802518 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:05.812364 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:05.812745 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:05.812922 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:05.817313 22507 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:37871
I20260812 00:10:05.817340 22576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:37871 every 8 connection(s)
I20260812 00:10:05.817721 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:05.818621 22577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:05.821606 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c: Bootstrap starting.
I20260812 00:10:05.822214 22577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:05.822562 22577 log.cc:824] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:05.823345 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c: No bootstrap required, opened a new log
I20260812 00:10:05.823920 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22507
I20260812 00:10:05.824043 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:05.824226 16783 external_mini_cluster.cc:1428] Setting key 4bee5ef00ed9a09c9402c76774361661
I20260812 00:10:05.825912 22577 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37871 } }
I20260812 00:10:05.826076 22577 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:05.826105 22577 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2152b08362b4ab3b2ad5639b09ca61c, State: Initialized, Role: FOLLOWER
I20260812 00:10:05.826246 22577 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37871 } }
I20260812 00:10:05.826392 22577 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:05.826490 22577 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:05.826560 22577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:05.827229 22577 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37871 } }
I20260812 00:10:05.827328 22577 leader_election.cc:302] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2152b08362b4ab3b2ad5639b09ca61c; no voters:
I20260812 00:10:05.827558 22577 leader_election.cc:288] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:05.827586 22582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:05.827698 22582 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [term 1 LEADER]: Becoming Leader. State: Replica: e2152b08362b4ab3b2ad5639b09ca61c, State: Running, Role: LEADER
I20260812 00:10:05.827800 22582 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37871 } }
I20260812 00:10:05.827852 22577 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:05.828132 22582 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37871 } } }
I20260812 00:10:05.828249 22582 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:05.828373 22582 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2152b08362b4ab3b2ad5639b09ca61c. Latest consensus state: current_term: 1 leader_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2152b08362b4ab3b2ad5639b09ca61c" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37871 } } }
I20260812 00:10:05.828439 22582 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:05.828590 22587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:05.829216 22587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:05.832403 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.824867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.16.99.254:37871)
0812 00:10:05.825146 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.825159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.825208 (+ 49us) server_negotiation.cc:407] Connection header received
0812 00:10:05.825923 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.825933 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.826034 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.826441 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:05.826672 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.828545 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.829877 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.830504 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.830656 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.831490 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:05.831495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:05.831495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:05.831710 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:05.831748 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:05.831761 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:05.831849 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:05.831932 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 00:10:05.832046 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
I20260812 00:10:05.833755 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:10:05.838004 22587 catalog_manager.cc:1347] Generated new cluster ID: 9d25d6c29acf4bc786ea30625fcbe65b
I20260812 00:10:05.838101 22587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:05.849550 22587 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:05.850131 22587 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:05.854120 22587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2152b08362b4ab3b2ad5639b09ca61c: Generated new TSK 0
I20260812 00:10:05.854401 22587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:10:05.868655 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:05.864461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.16.99.254:37871)
0812 00:10:05.864517 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:05.864519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:05.864580 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:10:05.864671 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:05.864672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:05.864683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:05.864750 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:10:05.864930 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.865521 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.866783 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.867015 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.867090 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.867322 (+ 232us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:10:05.868412 (+ 1090us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 00:10:05.868562 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260812 00:10:05.868728 22581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 00:10:05.870756 22606 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:05.864314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37871 (local address 127.0.0.1:43166)
0812 00:10:05.864511 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:05.864524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:05.864600 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:05.864769 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:05.864770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:05.864772 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:10:05.864837 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:05.864840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.865618 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.865620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:05.866667 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:05.866671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:05.867121 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:05.867123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:05.867189 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:05.867196 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:10:05.868524 (+ 1328us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 00:10:05.870640 (+ 2116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.99.254:37871: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1880,"thread_start_us":118,"threads_started":1}
I20260812 00:10:05.871044 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22507
E20260812 00:10:05.881587 22607 webserver.cc:533] Webserver: sslize failed:
E20260812 00:10:05.881709 22607 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T00:10:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (235 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T00:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:05Z Disabled control of system clock
I20260812 00:10:05.922551 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42785
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:42785
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/testchainca.cert with env {}
W20260812 00:10:06.018082 22611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:06.018397 22611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:06.018428 22611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:06.019727 22611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:06.019774 22611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:06.019788 22611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:06.019799 22611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:06.019811 22611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 00:10:06.019824 22611 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 00:10:06.019835 22611 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 00:10:06.021407 22611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:42785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:06.021643 22611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:06.021847 22611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:06.027370 22619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:06.027323 22616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:06.027475 22611 server_base.cc:1034] running on GCE node
W20260812 00:10:06.027321 22617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:06.027911 22611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:06.028249 22611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:06.029434 22611 hybrid_clock.cc:648] HybridClock initialized: now 1786493406029411 us; error 45 us; skew 500 ppm
I20260812 00:10:06.030944 22611 webserver.cc:460] Webserver started at http://127.16.99.254:44961/ using document root <none> and password file <none>
I20260812 00:10:06.031229 22611 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:06.031283 22611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:06.031447 22611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:06.032622 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "53e7bc09998d46919a3a5186ff2b55c2"
format_stamp: "Formatted at 2026-08-12 00:10:06 on dist-test-slave-zpfg"
server_key: "f64c1b24791611b8613e3afd4219a529"
server_key_iv: "b0642e8038d873b2d7b77d35375b75f5"
server_key_version: "encryptionkey@0"
I20260812 00:10:06.033031 22611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "53e7bc09998d46919a3a5186ff2b55c2"
format_stamp: "Formatted at 2026-08-12 00:10:06 on dist-test-slave-zpfg"
server_key: "f64c1b24791611b8613e3afd4219a529"
server_key_iv: "b0642e8038d873b2d7b77d35375b75f5"
server_key_version: "encryptionkey@0"
I20260812 00:10:06.034614 22611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260812 00:10:06.035641 22625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:06.035897 22611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:06.035990 22611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "53e7bc09998d46919a3a5186ff2b55c2"
format_stamp: "Formatted at 2026-08-12 00:10:06 on dist-test-slave-zpfg"
server_key: "f64c1b24791611b8613e3afd4219a529"
server_key_iv: "b0642e8038d873b2d7b77d35375b75f5"
server_key_version: "encryptionkey@0"
I20260812 00:10:06.036055 22611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:06.051484 22611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:06.051805 22611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:06.051920 22611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:06.056463 22611 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:42785
I20260812 00:10:06.056499 22681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:42785 every 8 connection(s)
I20260812 00:10:06.056913 22611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:06.058003 22682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:06.058873 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22611
I20260812 00:10:06.058965 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:06.059216 16783 external_mini_cluster.cc:1428] Setting key dc66310e533c3b924b1410d768338f03
I20260812 00:10:06.061631 22682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2: Bootstrap starting.
I20260812 00:10:06.062465 22682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:06.062785 22682 log.cc:824] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:06.063661 22682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2: No bootstrap required, opened a new log
I20260812 00:10:06.065900 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:06.059870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.16.99.254:42785)
0812 00:10:06.060095 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:06.060105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:06.060159 (+ 54us) server_negotiation.cc:407] Connection header received
0812 00:10:06.060709 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:06.060719 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:06.060821 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:06.060981 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:06.061195 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:06.062570 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:06.063549 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:06.064151 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:06.064275 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:06.065014 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:06.065018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:06.065018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:06.065230 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:06.065282 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:06.065298 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:06.065371 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:06.065526 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 00:10:06.065632 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260812 00:10:06.066110 22682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7bc09998d46919a3a5186ff2b55c2" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42785 } }
I20260812 00:10:06.066275 22682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:06.066334 22682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e7bc09998d46919a3a5186ff2b55c2, State: Initialized, Role: FOLLOWER
I20260812 00:10:06.066526 22682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7bc09998d46919a3a5186ff2b55c2" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42785 } }
I20260812 00:10:06.066633 22682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:06.066681 22682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:06.066725 22682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:06.067109 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:10:06.067418 22682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7bc09998d46919a3a5186ff2b55c2" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42785 } }
I20260812 00:10:06.067529 22682 leader_election.cc:302] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53e7bc09998d46919a3a5186ff2b55c2; no voters:
I20260812 00:10:06.067709 22682 leader_election.cc:288] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:06.067749 22687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:06.067914 22687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [term 1 LEADER]: Becoming Leader. State: Replica: 53e7bc09998d46919a3a5186ff2b55c2, State: Running, Role: LEADER
I20260812 00:10:06.067981 22682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:06.068050 22687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7bc09998d46919a3a5186ff2b55c2" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42785 } }
I20260812 00:10:06.068456 22687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53e7bc09998d46919a3a5186ff2b55c2. Latest consensus state: current_term: 1 leader_uuid: "53e7bc09998d46919a3a5186ff2b55c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7bc09998d46919a3a5186ff2b55c2" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42785 } } }
I20260812 00:10:06.068728 22687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:06.068635 22689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53e7bc09998d46919a3a5186ff2b55c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53e7bc09998d46919a3a5186ff2b55c2" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42785 } } }
I20260812 00:10:06.069144 22697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:06.069658 22697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:06.069170 22689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:06.071429 22697 catalog_manager.cc:1347] Generated new cluster ID: 3af8188917a34446b1c7ff76c265480a
I20260812 00:10:06.071477 22697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:06.086161 22697 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:06.086881 22697 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:06.089809 22697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53e7bc09998d46919a3a5186ff2b55c2: Generated new TSK 0
I20260812 00:10:06.090049 22697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:10:06.974506 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:06.100070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.16.99.254:42785)
0812 00:10:06.100141 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:06.100143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:06.100288 (+ 145us) server_negotiation.cc:407] Connection header received
0812 00:10:06.100378 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:06.100379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:06.100391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:06.100446 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0812 00:10:06.100619 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:06.101190 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:06.102423 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:06.102655 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:06.102713 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:06.102920 (+ 207us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 00:10:06.974186 (+871266us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 00:10:06.974442 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58458: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 00:10:06.974601 22685 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58458: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 00:10:06.974802 22712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:06.099996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42785 (local address 127.0.0.1:58458)
0812 00:10:06.100208 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:06.100220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:06.100296 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:06.100468 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:06.100469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:06.100471 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 00:10:06.100534 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:06.100537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:06.101291 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:06.101293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:06.102294 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:06.102297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:06.102743 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:06.102744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:06.102816 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:06.102823 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 00:10:06.974427 (+871604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 00:10:06.974573 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.99.254:42785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":126,"threads_started":1}
I20260812 00:10:06.975196 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22611
2026-08-12T00:10:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1082 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (98 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (105 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:07 dist-test-slave-zpfg krb5kdc[22724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:07 dist-test-slave-zpfg krb5kdc[22724](info): set up 2 sockets
Aug 12 00:10:07 dist-test-slave-zpfg krb5kdc[22724](info): commencing operation
krb5kdc: starting...
W20260812 00:10:08.314823 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:08 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:10:08.458601 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45007
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:08.543493 22738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:08.543781 22738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:08.543823 22738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:08.545140 22738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:08.545224 22738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:08.545248 22738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:08.545272 22738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:08.545333 22738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:08.547005 22738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:45007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:08.547322 22738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:08.547570 22738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:10:08.552945 22738 server_base.cc:1034] running on GCE node
W20260812 00:10:08.552884 22744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:08.552862 22746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:08.552912 22743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:08.553395 22738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:08.553679 22738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:08.554836 22738 hybrid_clock.cc:648] HybridClock initialized: now 1786493408554809 us; error 41 us; skew 500 ppm
Aug 12 00:10:08 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493408, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:08.557679 22738 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:10:08.558240 22738 webserver.cc:460] Webserver started at http://127.16.99.254:44057/ using document root <none> and password file <none>
I20260812 00:10:08.558557 22738 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:08.558629 22738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:08.558781 22738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:08.559937 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "03f3bf66ee3a4bfb827e0851a427e8ea"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "1bb32216833cd162c14ee577b4b89886"
server_key_iv: "a29552b660cfacbec78470bd33586995"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.560335 22738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "03f3bf66ee3a4bfb827e0851a427e8ea"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "1bb32216833cd162c14ee577b4b89886"
server_key_iv: "a29552b660cfacbec78470bd33586995"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.561774 22738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:08.562608 22753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:08.562805 22738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:08.562912 22738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "03f3bf66ee3a4bfb827e0851a427e8ea"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "1bb32216833cd162c14ee577b4b89886"
server_key_iv: "a29552b660cfacbec78470bd33586995"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.562987 22738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:08.577019 22738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:08.579568 22738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:08.579751 22738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:08.584272 22738 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:45007
I20260812 00:10:08.584355 22807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:45007 every 8 connection(s)
I20260812 00:10:08.584728 22738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:08.585669 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:08.588259 22808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea: Bootstrap starting.
I20260812 00:10:08.588838 22808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:08.589092 22808 log.cc:824] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:08.589838 22808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea: No bootstrap required, opened a new log
I20260812 00:10:08.591539 22808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45007 } }
I20260812 00:10:08.591687 22808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:08.591711 22808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f3bf66ee3a4bfb827e0851a427e8ea, State: Initialized, Role: FOLLOWER
I20260812 00:10:08.591827 22808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45007 } }
I20260812 00:10:08.591881 22808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:08.591902 22808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:08.591981 22808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:08.592566 22808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45007 } }
I20260812 00:10:08.592691 22808 leader_election.cc:302] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03f3bf66ee3a4bfb827e0851a427e8ea; no voters:
I20260812 00:10:08.592931 22808 leader_election.cc:288] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:08.592990 22811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:08.593223 22808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:08.593394 22811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [term 1 LEADER]: Becoming Leader. State: Replica: 03f3bf66ee3a4bfb827e0851a427e8ea, State: Running, Role: LEADER
I20260812 00:10:08.593494 22811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45007 } }
I20260812 00:10:08.593910 22813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03f3bf66ee3a4bfb827e0851a427e8ea. Latest consensus state: current_term: 1 leader_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45007 } } }
I20260812 00:10:08.593915 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f3bf66ee3a4bfb827e0851a427e8ea" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45007 } } }
I20260812 00:10:08.594012 22813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:08.594017 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:08.594249 22821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:08.594848 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22738
I20260812 00:10:08.594969 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:08.594987 22821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:08.595204 16783 external_mini_cluster.cc:1428] Setting key 3199083ca916fb48eb64cf5d9e92b2ac
I20260812 00:10:08.596726 22821 catalog_manager.cc:1347] Generated new cluster ID: 2dcea618b86f4150a7659adb3e48dd34
I20260812 00:10:08.596781 22821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:10:08 dist-test-slave-zpfg krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:08.607901 22821 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:08.608458 22821 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:08.608489 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:08.595733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.16.99.254:45007)
0812 00:10:08.595889 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:08.595898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:08.596281 (+ 383us) server_negotiation.cc:407] Connection header received
0812 00:10:08.596446 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:08.596455 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:08.596514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:08.596669 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:08.597612 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.599112 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.600225 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.600865 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.600963 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:08.604070 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:08.604075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:08.604076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:08.604090 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:08.606259 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:08.606702 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:08.606705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:08.606706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:08.606832 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:08.607052 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:08.607053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:08.607053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:08.607418 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:08.607597 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:08.608066 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 00:10:08.608214 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260812 00:10:08.611277 22821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03f3bf66ee3a4bfb827e0851a427e8ea: Generated new TSK 0
I20260812 00:10:08.611474 22821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:10:08.645527 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45007
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:08.741577 22835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:08.741865 22835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:08.741915 22835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:08.743331 22835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:08.743419 22835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:08.743468 22835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:08.745153 22835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:08.745462 22835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:08.745724 22835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:08.751389 22841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:08.751389 22840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:08.751603 22835 server_base.cc:1034] running on GCE node
W20260812 00:10:08.751412 22843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:08.752012 22835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:08.752310 22835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:08.753484 22835 hybrid_clock.cc:648] HybridClock initialized: now 1786493408753474 us; error 34 us; skew 500 ppm
Aug 12 00:10:08 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493408, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:08.756280 22835 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:10:08.756767 22835 webserver.cc:460] Webserver started at http://127.16.99.193:43947/ using document root <none> and password file <none>
I20260812 00:10:08.756961 22835 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:08.757001 22835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:08.757102 22835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:08.758141 22835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "9ab987c7bd4344b9b5af6200215a211a"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "58ac3e56990f0c392d49987bdb1d37c6"
server_key_iv: "8448eb5ca029a936851dd31a00da059f"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.758545 22835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "9ab987c7bd4344b9b5af6200215a211a"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "58ac3e56990f0c392d49987bdb1d37c6"
server_key_iv: "8448eb5ca029a936851dd31a00da059f"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.759861 22835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:08.760735 22850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:08.760950 22835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:08.761050 22835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "9ab987c7bd4344b9b5af6200215a211a"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "58ac3e56990f0c392d49987bdb1d37c6"
server_key_iv: "8448eb5ca029a936851dd31a00da059f"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.761123 22835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:08.786120 22835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:08.789105 22835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:08.789318 22835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:08.789604 22835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:08.789983 22835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:08.790045 22835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:08.790089 22835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:08.790122 22835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:08.797247 22835 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:41219
I20260812 00:10:08.797329 22967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:41219 every 8 connection(s)
I20260812 00:10:08.797734 22835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:08.804386 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22835
I20260812 00:10:08.804517 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:08.804750 16783 external_mini_cluster.cc:1428] Setting key 7286147cb32526130763b251f1371dec
Aug 12 00:10:08 dist-test-slave-zpfg krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493408, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:08.809489 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:08.798635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:39399 (local address 127.16.99.254:45007)
0812 00:10:08.798711 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:08.798715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:08.799029 (+ 314us) server_negotiation.cc:407] Connection header received
0812 00:10:08.800033 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:08.800034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:08.800044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:08.800102 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:08.800762 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.801271 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.802672 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.802880 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.802938 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:08.805779 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:08.805781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:08.805781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:08.805826 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:08.807588 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:08.808235 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:08.808236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:08.808237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:08.808294 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:08.808634 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:08.808635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:08.808635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:08.808742 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:08.808800 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:08.809236 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 00:10:08.809344 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:10:08.809520 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:08.798617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45007 (local address 127.16.99.193:39399)
0812 00:10:08.798861 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:08.798920 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:08.799935 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:08.800163 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:08.800167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:08.800467 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:08.800682 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:08.800687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.801461 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.801462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:08.802588 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:08.802592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.802982 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.802983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:08.803104 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:08.803813 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:08.803825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:08.805665 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:08.807696 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:08.807707 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:08.807709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:08.808139 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:08.808366 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:08.808367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:08.808367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:08.808562 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:08.808816 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:08.808823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:08.809005 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 00:10:08.809159 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 00:10:08.809319 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":141,"threads_started":1}
I20260812 00:10:08.810200 22968 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45007
I20260812 00:10:08.810336 22968 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:08.810598 22968 heartbeater.cc:511] Master 127.16.99.254:45007 requested a full tablet report, sending...
I20260812 00:10:08.811183 22770 ts_manager.cc:194] Registered new tserver with Master: 9ab987c7bd4344b9b5af6200215a211a (127.16.99.193:41219)
I20260812 00:10:08.811888 22770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:39399
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:10:08.838133 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45007
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:08.925308 22975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:08.925560 22975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:08.925588 22975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:08.926923 22975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:08.926980 22975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:08.927011 22975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:08.928591 22975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:08.928814 22975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:08.929016 22975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:08.934636 22983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:08.934669 22981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:08.934636 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:08.935037 22975 server_base.cc:1034] running on GCE node
I20260812 00:10:08.935307 22975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:08.935588 22975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:08.936815 22975 hybrid_clock.cc:648] HybridClock initialized: now 1786493408936725 us; error 65 us; skew 500 ppm
Aug 12 00:10:08 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493408, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:08.940311 22975 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:10:08.940881 22975 webserver.cc:460] Webserver started at http://127.16.99.194:33109/ using document root <none> and password file <none>
I20260812 00:10:08.941092 22975 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:08.941138 22975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:08.941237 22975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:08.942291 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "872726b34a544f8cae0ba1fc6764f0df"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "4b6f1974d6393c44ade521021464cd6d"
server_key_iv: "5b3470e15f6b1dd4efb70ed5a5b465df"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.942703 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "872726b34a544f8cae0ba1fc6764f0df"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "4b6f1974d6393c44ade521021464cd6d"
server_key_iv: "5b3470e15f6b1dd4efb70ed5a5b465df"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.944202 22975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:08.945168 22990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:08.945431 22975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:08.945513 22975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "872726b34a544f8cae0ba1fc6764f0df"
format_stamp: "Formatted at 2026-08-12 00:10:08 on dist-test-slave-zpfg"
server_key: "4b6f1974d6393c44ade521021464cd6d"
server_key_iv: "5b3470e15f6b1dd4efb70ed5a5b465df"
server_key_version: "encryptionkey@0"
I20260812 00:10:08.945617 22975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:08.964609 22975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:08.967232 22975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:08.967417 22975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:08.967684 22975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:08.968027 22975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:08.968086 22975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:08.968130 22975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:08.968155 22975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:08.974370 22975 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:33511
I20260812 00:10:08.974433 23111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:33511 every 8 connection(s)
I20260812 00:10:08.974799 22975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:10:08 dist-test-slave-zpfg krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493408, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:08.984728 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 22975
I20260812 00:10:08.984867 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:08.985064 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:08.975712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:39963 (local address 127.16.99.254:45007)
0812 00:10:08.975847 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:08.975850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:08.975987 (+ 137us) server_negotiation.cc:407] Connection header received
0812 00:10:08.977058 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:08.977060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:08.977078 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:08.977145 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:08.977820 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.978323 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.979413 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.979614 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.979693 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:08.982419 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:08.982420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:08.982421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:08.982429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:08.983639 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:08.984021 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:08.984022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:08.984022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:08.984074 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:08.984332 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:08.984332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:08.984333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:08.984442 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:08.984496 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:08.984867 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 00:10:08.984962 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 00:10:08.985074 16783 external_mini_cluster.cc:1428] Setting key 6145335efc13166e87cf0b283e4ee747
I20260812 00:10:08.985395 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:08.975583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45007 (local address 127.16.99.194:39963)
0812 00:10:08.975833 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:08.975886 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:08.976938 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:08.977223 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:08.977227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:08.977523 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:08.977719 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:08.977723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.978463 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.978464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:08.979306 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:08.979308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:08.979701 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:08.979702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:08.979839 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:08.980479 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:08.980487 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:08.982268 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:08.983720 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:08.983727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:08.983729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:08.983952 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:08.984144 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:08.984145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:08.984145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:08.984265 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:08.984550 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:08.984557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:08.984694 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 00:10:08.984805 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 00:10:08.985035 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260812 00:10:08.985981 23112 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45007
I20260812 00:10:08.986073 23112 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:08.986279 23112 heartbeater.cc:511] Master 127.16.99.254:45007 requested a full tablet report, sending...
I20260812 00:10:08.986784 22770 ts_manager.cc:194] Registered new tserver with Master: 872726b34a544f8cae0ba1fc6764f0df (127.16.99.194:33511)
I20260812 00:10:08.987367 22770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:39963
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:10:09.018850 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45007
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:09.113437 23118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:09.113700 23118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:09.113730 23118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:09.115227 23118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:09.115279 23118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:09.115312 23118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:09.117084 23118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:09.117393 23118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:09.117671 23118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:09.123672 23128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:09.123670 23124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:09.123670 23125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:09.124162 23118 server_base.cc:1034] running on GCE node
I20260812 00:10:09.124477 23118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:09.124754 23118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:09.125929 23118 hybrid_clock.cc:648] HybridClock initialized: now 1786493409125899 us; error 52 us; skew 500 ppm
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.128811 23118 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:10:09.129305 23118 webserver.cc:460] Webserver started at http://127.16.99.195:46663/ using document root <none> and password file <none>
I20260812 00:10:09.129522 23118 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:09.129565 23118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:09.129660 23118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:09.130843 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "f2869fcb800c4c21a139bc7c73f7932f"
format_stamp: "Formatted at 2026-08-12 00:10:09 on dist-test-slave-zpfg"
server_key: "099440f115cd5cabd64aa93e54184bbf"
server_key_iv: "1cafcabd18ed702d873b99031f6a671e"
server_key_version: "encryptionkey@0"
I20260812 00:10:09.131173 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "f2869fcb800c4c21a139bc7c73f7932f"
format_stamp: "Formatted at 2026-08-12 00:10:09 on dist-test-slave-zpfg"
server_key: "099440f115cd5cabd64aa93e54184bbf"
server_key_iv: "1cafcabd18ed702d873b99031f6a671e"
server_key_version: "encryptionkey@0"
I20260812 00:10:09.132565 23118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:10:09.133450 23135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:09.133852 23118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:09.133978 23118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "f2869fcb800c4c21a139bc7c73f7932f"
format_stamp: "Formatted at 2026-08-12 00:10:09 on dist-test-slave-zpfg"
server_key: "099440f115cd5cabd64aa93e54184bbf"
server_key_iv: "1cafcabd18ed702d873b99031f6a671e"
server_key_version: "encryptionkey@0"
I20260812 00:10:09.134151 23118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:09.144513 23118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:09.147326 23118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:09.147540 23118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:09.147836 23118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:09.148244 23118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:09.148303 23118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:09.148329 23118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:09.148365 23118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:09.156442 23118 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:38675
I20260812 00:10:09.156489 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:38675 every 8 connection(s)
I20260812 00:10:09.156863 23118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:09.165149 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 23118
I20260812 00:10:09.165243 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:09.165462 16783 external_mini_cluster.cc:1428] Setting key 23be6adb3fe77681fc6083147e326195
I20260812 00:10:09.168889 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:09.157693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58181 (local address 127.16.99.254:45007)
0812 00:10:09.157843 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:09.157847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:09.158057 (+ 210us) server_negotiation.cc:407] Connection header received
0812 00:10:09.159349 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:09.159350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:09.159365 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:09.159446 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:09.160232 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.160796 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.161953 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.162192 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.162285 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:09.165012 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:09.165014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:09.165014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:09.165024 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:09.167162 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.167658 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.167659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.167659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.167716 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.168047 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.168048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.168048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.168306 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:09.168391 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:09.168721 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 00:10:09.168814 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 00:10:09.169050 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:09.157653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45007 (local address 127.16.99.195:58181)
0812 00:10:09.157899 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:09.157959 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:09.159195 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:09.159538 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:09.159542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:09.159896 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:09.160114 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:09.160119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.160909 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.160910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:09.161844 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:09.161846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.162296 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.162297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:09.162482 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:09.163120 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:09.163130 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:09.164900 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:09.167315 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:09.167323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:09.167325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:09.167586 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:09.167797 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:09.167798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:09.167798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:09.167923 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:09.168419 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:09.168426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:09.168565 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 00:10:09.168661 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 00:10:09.168812 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260812 00:10:09.169610 23254 heartbeater.cc:347] Connected to a master server at 127.16.99.254:45007
I20260812 00:10:09.169706 23254 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:09.169934 23254 heartbeater.cc:511] Master 127.16.99.254:45007 requested a full tablet report, sending...
I20260812 00:10:09.170461 22770 ts_manager.cc:194] Registered new tserver with Master: f2869fcb800c4c21a139bc7c73f7932f (127.16.99.195:38675)
I20260812 00:10:09.170928 22770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:58181
I20260812 00:10:09.177994 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/test-admin.keytab.
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.197013 16783 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 00:10:09.199203 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:09.198165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.16.99.193:41219)
0812 00:10:09.198459 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:09.198463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:09.198478 (+ 15us) server_negotiation.cc:407] Connection header received
0812 00:10:09.198506 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:09.198507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:09.198524 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:09.198599 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:09.199118 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50204: BlockingRecv error: recv got EOF from 127.0.0.1:50204 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
W20260812 00:10:09.199548 16783 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.200920 16783 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.202078 16783 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:10:09.211265 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:09.202780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.16.99.193:41219)
0812 00:10:09.202917 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:09.202920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:09.203009 (+ 89us) server_negotiation.cc:407] Connection header received
0812 00:10:09.203090 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:09.203091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:09.203111 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:09.203178 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:09.203634 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.204488 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.205229 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.205592 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.205674 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:09.208168 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:09.208177 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:09.208177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:09.208185 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:09.209775 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.210244 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.210247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.210247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.210309 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.210653 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.210654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.210655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.210815 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:09.210930 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:09.211017 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 00:10:09.211096 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260812 00:10:09.213419 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:09.212771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.16.99.193:41219)
0812 00:10:09.212822 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:09.212824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:09.212875 (+ 51us) server_negotiation.cc:407] Connection header received
0812 00:10:09.212948 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:09.212949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:09.212973 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:09.213035 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:09.213363 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50210: BlockingRecv error: recv got EOF from 127.0.0.1:50210 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260812 00:10:09.213529 16783 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.214630 16783 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.215493 16783 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:10:09.223253 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:09.216126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.16.99.193:41219)
0812 00:10:09.216187 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:09.216189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:09.216198 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:09.216294 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:09.216295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:09.216304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:09.216356 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:09.216857 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.217370 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.218123 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.218525 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.218579 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:09.220707 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:09.220708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:09.220708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:09.220715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:09.222164 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.222559 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.222560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.222561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.222611 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.222778 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.222779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.222779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.222886 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:09.222958 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:09.223044 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 00:10:09.223107 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260812 00:10:09.224893 23261 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:09.224326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.16.99.193:41219)
0812 00:10:09.224366 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:09.224367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:09.224498 (+ 131us) server_negotiation.cc:407] Connection header received
0812 00:10:09.224532 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:09.224533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:09.224541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:09.224585 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:09.224849 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50228: BlockingRecv error: recv got EOF from 127.0.0.1:50228 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 00:10:09.224998 16783 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786493381608766-16783-0/krb5kdc/krb5cc' of type FILE
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.226001 16783 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:09.226915 16783 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[22724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:10:09.234417 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:09.227439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.16.99.193:41219)
0812 00:10:09.227495 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:09.227496 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:09.227526 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:10:09.227588 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:09.227589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:09.227597 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:09.227643 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:09.227959 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.228425 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.229097 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:09.229277 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:09.229328 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:09.231539 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:09.231540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:09.231541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:09.231549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:09.233209 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.233552 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.233553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.233553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.233607 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:09.233769 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:09.233770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:09.233770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:09.233879 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:09.234000 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:09.234211 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 00:10:09.234296 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:10:09.235100 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22835
I20260812 00:10:09.242715 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22975
I20260812 00:10:09.249346 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 23118
I20260812 00:10:09.255935 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 22738
2026-08-12T00:10:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2079 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[23275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[23275](info): set up 2 sockets
Aug 12 00:10:09 dist-test-slave-zpfg krb5kdc[23275](info): commencing operation
krb5kdc: starting...
W20260812 00:10:10.342154 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:10 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab.
Entry for principal oryx/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab.
I20260812 00:10:10.482097 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40411
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab
--principal=oryx/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:40411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:10.566934 23288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:10.567180 23288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:10.567204 23288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:10.568447 23288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:10.568498 23288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:10.568513 23288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:10.568529 23288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:10.568542 23288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:10.570120 23288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:40411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab
--principal=oryx/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:10.570415 23288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:10.570670 23288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:10.576321 23293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:10.576385 23296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:10.576468 23288 server_base.cc:1034] running on GCE node
W20260812 00:10:10.576278 23294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:10.576769 23288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:10.577116 23288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:10.578290 23288 hybrid_clock.cc:648] HybridClock initialized: now 1786493410578269 us; error 43 us; skew 500 ppm
Aug 12 00:10:10 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:10.581137 23288 init.cc:377] Logged in from keytab as oryx/127.16.99.254@KRBTEST.COM (short username oryx)
I20260812 00:10:10.581764 23288 webserver.cc:460] Webserver started at http://127.16.99.254:42955/ using document root <none> and password file <none>
I20260812 00:10:10.582055 23288 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:10.582130 23288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:10.582297 23288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:10.583451 23288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "d7225955d25c46e8a0d7b98372f77300"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "70860822b61bf42385653f4d9d408fcc"
server_key_iv: "81a75bf195390af536e345de90c1d6ef"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.583842 23288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "d7225955d25c46e8a0d7b98372f77300"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "70860822b61bf42385653f4d9d408fcc"
server_key_iv: "81a75bf195390af536e345de90c1d6ef"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.585258 23288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:10:10.586090 23304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:10.586287 23288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:10.586418 23288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "d7225955d25c46e8a0d7b98372f77300"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "70860822b61bf42385653f4d9d408fcc"
server_key_iv: "81a75bf195390af536e345de90c1d6ef"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.586514 23288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:10.603333 23288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:10.606020 23288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:10.606237 23288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:10.610795 23288 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:40411
I20260812 00:10:10.610863 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:40411 every 8 connection(s)
I20260812 00:10:10.611254 23288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:10.612425 23359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:10.615029 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: Bootstrap starting.
I20260812 00:10:10.615700 23359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:10.616042 23359 log.cc:824] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:10.616930 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: No bootstrap required, opened a new log
I20260812 00:10:10.618310 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 23288
I20260812 00:10:10.618467 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:10.618681 16783 external_mini_cluster.cc:1428] Setting key 5aac22089c31de09af4f1567b76aa5e6
I20260812 00:10:10.619185 23359 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } }
I20260812 00:10:10.619418 23359 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:10.619495 23359 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7225955d25c46e8a0d7b98372f77300, State: Initialized, Role: FOLLOWER
I20260812 00:10:10.619644 23359 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } }
I20260812 00:10:10.619757 23359 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:10.619817 23359 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:10.619876 23359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:10.621198 23359 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } }
I20260812 00:10:10.621340 23359 leader_election.cc:302] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7225955d25c46e8a0d7b98372f77300; no voters:
I20260812 00:10:10.621569 23359 leader_election.cc:288] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:10.621601 23364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:10.621724 23364 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 1 LEADER]: Becoming Leader. State: Replica: d7225955d25c46e8a0d7b98372f77300, State: Running, Role: LEADER
I20260812 00:10:10.621829 23364 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } }
I20260812 00:10:10.621877 23359 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:10.622179 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } }
I20260812 00:10:10.622385 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7225955d25c46e8a0d7b98372f77300. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } }
I20260812 00:10:10.622964 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:10.622975 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:10.623275 23376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:10.623798 23376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:10.625528 23376 catalog_manager.cc:1347] Generated new cluster ID: 6322e762f280494b82b3a368b222fb97
I20260812 00:10:10.625593 23376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:10:10 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:10.631477 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:10.619343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.16.99.254:40411)
0812 00:10:10.619613 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:10.619623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:10.619658 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:10:10.619794 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:10.619803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:10.619863 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:10.620011 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:10.620930 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:10.622211 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:10.623532 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:10.624135 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:10.624255 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:10.627376 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:10.627381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:10.627381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:10.627396 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:10.629638 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:10.630041 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:10.630045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:10.630045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:10.630199 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:10.630431 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:10.630432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:10.630432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:10.630777 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:10.630963 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:10.631061 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 00:10:10.631220 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
I20260812 00:10:10.640764 23376 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:10.641382 23376 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:10.645695 23376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: Generated new TSK 0
I20260812 00:10:10.645912 23376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab.
Entry for principal oryx/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab.
I20260812 00:10:10.682740 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40411
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab
--principal=oryx/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:10.776393 23387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:10.776751 23387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:10.776796 23387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:10.778833 23387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:10.778909 23387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:10.778957 23387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:10.780879 23387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab
--principal=oryx/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:10.781191 23387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:10.781455 23387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:10.786856 23392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:10.786880 23395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:10.786899 23393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:10.787187 23387 server_base.cc:1034] running on GCE node
I20260812 00:10:10.787535 23387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:10.787897 23387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:10.789080 23387 hybrid_clock.cc:648] HybridClock initialized: now 1786493410788991 us; error 113 us; skew 500 ppm
Aug 12 00:10:10 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:10.791985 23387 init.cc:377] Logged in from keytab as oryx/127.16.99.193@KRBTEST.COM (short username oryx)
I20260812 00:10:10.792506 23387 webserver.cc:460] Webserver started at http://127.16.99.193:33047/ using document root <none> and password file <none>
I20260812 00:10:10.792706 23387 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:10.792748 23387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:10.792845 23387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:10.794215 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "ffa32349f97846248d6f3b2faa2ed898"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "18870cead4c946ba649ec1c8e5f6546f"
server_key_iv: "6cfdc063b86b692c817bf8a2dcd913d9"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.794744 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "ffa32349f97846248d6f3b2faa2ed898"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "18870cead4c946ba649ec1c8e5f6546f"
server_key_iv: "6cfdc063b86b692c817bf8a2dcd913d9"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.796844 23387 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:10.798038 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:10.798321 23387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:10.798468 23387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "ffa32349f97846248d6f3b2faa2ed898"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "18870cead4c946ba649ec1c8e5f6546f"
server_key_iv: "6cfdc063b86b692c817bf8a2dcd913d9"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.798560 23387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:10.821888 23387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:10.825215 23387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:10.825403 23387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:10.825681 23387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:10.826035 23387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:10.826093 23387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:10.826139 23387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:10.826159 23387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:10.834046 23387 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:40277
I20260812 00:10:10.834095 23519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:40277 every 8 connection(s)
I20260812 00:10:10.834566 23387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:10.839540 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 23387
I20260812 00:10:10.839673 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:10.839882 16783 external_mini_cluster.cc:1428] Setting key 32ad26c0fee36c904eb4ebe2cfdc7e45
Aug 12 00:10:10 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:10.847816 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:10.835753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:42177 (local address 127.16.99.254:40411)
0812 00:10:10.835811 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:10.835813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:10.835825 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:10.836705 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:10.836707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:10.836725 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:10.836807 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:10.838409 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:10.839282 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:10.841022 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:10.841343 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:10.841423 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:10.844358 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:10.844360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:10.844361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:10.844369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:10.846078 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:10.846705 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:10.846707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:10.846707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:10.846767 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:10.847087 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:10.847087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:10.847087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:10.847205 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:10.847257 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:10.847652 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 00:10:10.847744 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:10:10.848129 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:10.835451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.193:42177)
0812 00:10:10.835642 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:10.835707 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:10.836559 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:10.836900 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:10.836905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:10.837868 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:10.838226 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:10.838236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:10.839416 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:10.839417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:10.840923 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:10.840926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:10.841452 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:10.841454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:10.841636 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:10.842332 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:10.842395 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:10.844239 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:10.846172 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:10.846182 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:10.846185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:10.846610 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:10.846848 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:10.846849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:10.846849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:10.847014 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:10.847276 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:10.847283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:10.847483 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 00:10:10.847624 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 00:10:10.847853 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260812 00:10:10.848812 23520 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40411
I20260812 00:10:10.848912 23520 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:10.849128 23520 heartbeater.cc:511] Master 127.16.99.254:40411 requested a full tablet report, sending...
I20260812 00:10:10.849803 23322 ts_manager.cc:194] Registered new tserver with Master: ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
WARNING: no policy specified for oryx/127.16.99.194@KRBTEST.COM; defaulting to no policy
I20260812 00:10:10.850878 23322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:42177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab.
Entry for principal oryx/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab.
I20260812 00:10:10.871349 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40411
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab
--principal=oryx/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:10.957794 23525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:10.958032 23525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:10.958060 23525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:10.959347 23525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:10.959400 23525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:10.959429 23525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:10.960983 23525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab
--principal=oryx/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:10.961272 23525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:10.961532 23525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:10.967545 23533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:10.967554 23525 server_base.cc:1034] running on GCE node
W20260812 00:10:10.967535 23531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:10.967514 23530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:10.968041 23525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:10.968364 23525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:10.969542 23525 hybrid_clock.cc:648] HybridClock initialized: now 1786493410969536 us; error 53 us; skew 500 ppm
Aug 12 00:10:10 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:10.972383 23525 init.cc:377] Logged in from keytab as oryx/127.16.99.194@KRBTEST.COM (short username oryx)
I20260812 00:10:10.972905 23525 webserver.cc:460] Webserver started at http://127.16.99.194:46841/ using document root <none> and password file <none>
I20260812 00:10:10.973145 23525 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:10.973192 23525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:10.973337 23525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:10.974530 23525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "b376fa291da84a088858823be12e036d"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "851387cba72752123aa537d01db8f189"
server_key_iv: "fa8f9fba7850bc6cc1457965d16862c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.974893 23525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "b376fa291da84a088858823be12e036d"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "851387cba72752123aa537d01db8f189"
server_key_iv: "fa8f9fba7850bc6cc1457965d16862c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.976394 23525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:10:10.977299 23540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:10.977536 23525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:10.977645 23525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "b376fa291da84a088858823be12e036d"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "851387cba72752123aa537d01db8f189"
server_key_iv: "fa8f9fba7850bc6cc1457965d16862c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:10.977725 23525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:10.994977 23525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:10.997231 23525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:10.997440 23525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:10.997725 23525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:10.998070 23525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:10.998124 23525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:10.998160 23525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:10.998207 23525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.005067 23525 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:44107
I20260812 00:10:11.005121 23660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:44107 every 8 connection(s)
I20260812 00:10:11.005518 23525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:10:11.007741 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 23525
I20260812 00:10:11.007891 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:11.008116 16783 external_mini_cluster.cc:1428] Setting key af39ade18d0d7838108f1dfa3792dba3
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.99.195@KRBTEST.COM; defaulting to no policy
I20260812 00:10:11.017299 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.006540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:36727 (local address 127.16.99.254:40411)
0812 00:10:11.006623 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.006625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.006875 (+ 250us) server_negotiation.cc:407] Connection header received
0812 00:10:11.007497 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.007499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.007512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.007571 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.008757 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.009623 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.010801 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.011069 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.011130 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.013905 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.013907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.013907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.013917 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.015592 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.016131 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.016133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.016134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.016204 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.016529 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.016530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.016530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.016664 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.016754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.017077 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.017180 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:10:11.017535 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.006530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.194:36727)
0812 00:10:11.006735 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.006776 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.007381 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.007651 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.007656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.008381 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.008650 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.008657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.009734 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.009735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.010692 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.010696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.011162 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.011163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.011387 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.012083 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.012093 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.013783 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.015713 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.015723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.015726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.016019 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.016292 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.016293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.016293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.016443 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.016743 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.016752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.016926 (+ 174us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.017054 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.017248 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260812 00:10:11.018081 23661 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40411
I20260812 00:10:11.018179 23661 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:11.018424 23661 heartbeater.cc:511] Master 127.16.99.254:40411 requested a full tablet report, sending...
I20260812 00:10:11.018909 23322 ts_manager.cc:194] Registered new tserver with Master: b376fa291da84a088858823be12e036d (127.16.99.194:44107)
I20260812 00:10:11.019421 23322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:36727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab.
Entry for principal oryx/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab.
I20260812 00:10:11.038028 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40411
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab
--principal=oryx/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:11.124660 23666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:11.124889 23666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:11.124915 23666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:11.126053 23666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:11.126096 23666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:11.126123 23666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:11.127717 23666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab
--principal=oryx/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:11.127948 23666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:11.128134 23666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:11.134024 23671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.134043 23666 server_base.cc:1034] running on GCE node
W20260812 00:10:11.134024 23672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.134038 23674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.134491 23666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:11.134747 23666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:11.135929 23666 hybrid_clock.cc:648] HybridClock initialized: now 1786493411135915 us; error 33 us; skew 500 ppm
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:11.139012 23666 init.cc:377] Logged in from keytab as oryx/127.16.99.195@KRBTEST.COM (short username oryx)
I20260812 00:10:11.139534 23666 webserver.cc:460] Webserver started at http://127.16.99.195:34801/ using document root <none> and password file <none>
I20260812 00:10:11.139869 23666 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.139928 23666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.140095 23666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:11.141291 23666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "ae1a1ad80ebf4d6fa74b3c6020662589"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "cd9ddfdd1f8054f5177e783fc8b50698"
server_key_iv: "92172d394c43d02ef49c49b41cd732fc"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.141700 23666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "ae1a1ad80ebf4d6fa74b3c6020662589"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "cd9ddfdd1f8054f5177e783fc8b50698"
server_key_iv: "92172d394c43d02ef49c49b41cd732fc"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.143405 23666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:11.144569 23681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.144824 23666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:11.144944 23666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "ae1a1ad80ebf4d6fa74b3c6020662589"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "cd9ddfdd1f8054f5177e783fc8b50698"
server_key_iv: "92172d394c43d02ef49c49b41cd732fc"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.145026 23666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.159134 23666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:11.161830 23666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:11.161965 23666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:11.162156 23666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:11.162587 23666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:11.162648 23666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.162699 23666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:11.162734 23666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.168697 23666 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:44283
I20260812 00:10:11.168800 23802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:44283 every 8 connection(s)
I20260812 00:10:11.169142 23666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:11.174558 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 23666
I20260812 00:10:11.174643 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:11.174862 16783 external_mini_cluster.cc:1428] Setting key e7b7f5f735aa7edf3d545215e29f2cb2
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:11.180823 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.170138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:37903 (local address 127.16.99.254:40411)
0812 00:10:11.170207 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.170209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.170470 (+ 261us) server_negotiation.cc:407] Connection header received
0812 00:10:11.171077 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.171079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.171093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.171154 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.172162 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.172676 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.173744 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.173950 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.174031 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.177434 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.177436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.177436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.177450 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.179044 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.179574 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.179575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.179575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.179639 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.179937 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.179938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.179939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.180069 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.180293 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.180598 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.180710 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:10:11.181133 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.170120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.195:37903)
0812 00:10:11.170283 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.170334 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.170926 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.171237 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.171240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.171847 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.172053 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.172058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.172796 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.172797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.173636 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.173638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.174041 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.174042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.174179 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.174953 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.174965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.177308 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.179172 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.179183 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.179186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.179470 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.179722 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.179723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.179724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.179858 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.180279 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.180288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.180452 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.180573 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.180823 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260812 00:10:11.181738 23803 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40411
I20260812 00:10:11.181928 23803 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:11.182129 23803 heartbeater.cc:511] Master 127.16.99.254:40411 requested a full tablet report, sending...
I20260812 00:10:11.182545 23322 ts_manager.cc:194] Registered new tserver with Master: ae1a1ad80ebf4d6fa74b3c6020662589 (127.16.99.195:44283)
I20260812 00:10:11.183009 23322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:37903
I20260812 00:10:11.187625 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.99.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.253.keytab.
Entry for principal oryx/127.16.99.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.253.keytab.
I20260812 00:10:11.218093 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:35763
--webserver_interface=127.16.99.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.253.keytab
--principal=oryx/127.16.99.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:40411,127.16.99.253:35763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:11.309902 23809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:11.310179 23809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:11.310223 23809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:11.311517 23809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:11.311594 23809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:11.311618 23809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:11.311640 23809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:11.311668 23809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:11.313267 23809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:40411,127.16.99.253:35763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.253.keytab
--principal=oryx/127.16.99.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.99.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
W20260812 00:10:11.313423 23809 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.99.254:40411,127.16.99.253:35763'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 00:10:11.313591 23809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:11.313850 23809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:11.319332 23816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.319327 23815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.319339 23818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.319546 23809 server_base.cc:1034] running on GCE node
I20260812 00:10:11.319883 23809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:11.320205 23809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:11.321378 23809 hybrid_clock.cc:648] HybridClock initialized: now 1786493411321342 us; error 56 us; skew 500 ppm
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:11.324359 23809 init.cc:377] Logged in from keytab as oryx/127.16.99.253@KRBTEST.COM (short username oryx)
I20260812 00:10:11.324932 23809 webserver.cc:460] Webserver started at http://127.16.99.253:46101/ using document root <none> and password file <none>
I20260812 00:10:11.325186 23809 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.325255 23809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.325407 23809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:11.326658 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/instance:
uuid: "9b3c555aad11484eb965ff159e1b9f25"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "8a0e45062e8d8af4677ab72a6d2ad05d"
server_key_iv: "cbf5de202cf4c5d8f1789e677d305762"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.327028 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal/instance:
uuid: "9b3c555aad11484eb965ff159e1b9f25"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "8a0e45062e8d8af4677ab72a6d2ad05d"
server_key_iv: "cbf5de202cf4c5d8f1789e677d305762"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.328438 23809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:11.329373 23825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.329666 23809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:11.329779 23809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
uuid: "9b3c555aad11484eb965ff159e1b9f25"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "8a0e45062e8d8af4677ab72a6d2ad05d"
server_key_iv: "cbf5de202cf4c5d8f1789e677d305762"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.329872 23809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.370373 23809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:11.373049 23809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:11.373226 23809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:11.377727 23809 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.253:35763
I20260812 00:10:11.377765 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.253:35763 every 8 connection(s)
I20260812 00:10:11.378198 23809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
I20260812 00:10:11.379508 23882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:11.381534 23882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:10:11.385685 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 23809
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.253@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.99.253@KRBTEST.COM
I20260812 00:10:11.394867 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.382703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.16.99.254:40411)
0812 00:10:11.382797 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.382800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.383259 (+ 459us) server_negotiation.cc:407] Connection header received
0812 00:10:11.384002 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.384003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.384018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.384074 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.384860 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.385416 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.387087 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.387436 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.387526 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.390633 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.390635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.390635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.390645 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.392762 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.393541 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.393542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.393542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.393658 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.394017 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.394018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.394019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.394191 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.394287 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.394649 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.394743 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 00:10:11.395126 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.382578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:59924)
0812 00:10:11.382888 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.382965 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.383873 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.384192 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.384196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.384494 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.384730 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.384736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.385591 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.385592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.386988 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.386990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.387579 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.387581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.387797 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.388421 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.388432 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.390519 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.392869 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.392880 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.392883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.393340 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.393799 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.393800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.393801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.393923 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.394278 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.394285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.394501 (+ 216us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.394615 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.394870 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":189,"threads_started":1}
I20260812 00:10:11.395792 23882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:10:11.397358 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.386373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.16.99.253:35763)
0812 00:10:11.386590 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.386595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.386622 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:10:11.386643 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.386645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.386666 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.386751 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.387600 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.388831 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.390035 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.390564 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.390645 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.394060 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.394071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.394071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.394079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.396014 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.396480 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.396483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.396484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.396548 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.396774 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.396775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.396775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.396926 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.397108 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.397153 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.397240 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":118,"threads_started":1}
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.253@KRBTEST.COM for oryx/127.16.99.253@KRBTEST.COM
I20260812 00:10:11.404330 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.396125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.0.0.1:58696)
0812 00:10:11.396182 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.396193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.396356 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.396506 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.396507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.396634 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.396682 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.396683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.397323 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.397324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.398095 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.398098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.398636 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.398637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.398693 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.399130 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.399135 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.400787 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.403078 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.403079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.403079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.403252 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.403543 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.403544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.403545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.403604 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.404013 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.404013 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.404056 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.404116 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.404206 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 00:10:11.404489 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.396191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.16.99.253:35763)
0812 00:10:11.396382 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.396383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.396392 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:11.396410 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.396411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.396420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.396488 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.396761 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.397247 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.398285 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.398573 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.398631 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.400955 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.400956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.400956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.400963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.402975 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.403405 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.403406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.403406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.403472 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.403802 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.403803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.403803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.403936 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.404005 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.404150 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.404336 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":168,"thread_start_us":114,"threads_started":1}
I20260812 00:10:11.405874 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Bootstrap starting.
I20260812 00:10:11.406594 23882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:11.406839 23882 log.cc:824] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:11.407658 23882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: No bootstrap required, opened a new log
I20260812 00:10:11.409096 23882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } }
I20260812 00:10:11.409227 23882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:11.409260 23882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3c555aad11484eb965ff159e1b9f25, State: Initialized, Role: FOLLOWER
I20260812 00:10:11.409391 23882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } }
I20260812 00:10:11.409648 23890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } } }
I20260812 00:10:11.409726 23882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:11.409739 23890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:10:11.411837 23902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:10:11.411906 23902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:10:11.413715 23809 master_runner.cc:428] Detected that this master 9b3c555aad11484eb965ff159e1b9f25 is joining an existing cluster
I20260812 00:10:11.413817 23809 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.99.253:35763
I20260812 00:10:11.422376 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.414877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:59930)
0812 00:10:11.415045 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.415056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.415143 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.415289 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.415289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.415538 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.415615 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.415618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.416637 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.416638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.417714 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.417717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.418220 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.418221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.418318 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.418783 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.418788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.419147 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.420990 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.420992 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.420992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.421296 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.421545 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.421546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.421546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.421634 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.421980 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.421980 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.422027 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.422111 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.422231 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":53,"thread_start_us":95,"threads_started":1}
I20260812 00:10:11.422362 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.414927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.16.99.254:40411)
0812 00:10:11.414979 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.414982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.415112 (+ 130us) server_negotiation.cc:407] Connection header received
0812 00:10:11.415205 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.415206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.415217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.415283 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.415699 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.416524 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.417818 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.418093 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.418189 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.419236 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.419238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.419238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.419253 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.420871 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.421399 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.421400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.421401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.421467 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.421713 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.421715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.421715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.421870 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.421928 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.422143 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.422230 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:10:11.422608 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.415096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.16.99.253:35763)
0812 00:10:11.415131 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.415133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.415207 (+ 74us) server_negotiation.cc:407] Connection header received
0812 00:10:11.415307 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.415308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.415318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.415349 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.415864 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.416577 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.417326 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.417566 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.417657 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.419028 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.419030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.419030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.419039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.421039 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.421478 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.421479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.421479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.421579 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.421799 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.421800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.421801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.421997 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.422126 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.422309 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.422434 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:10:11.423044 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.414929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.0.0.1:58700)
0812 00:10:11.415138 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.415149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.415233 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.415412 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.415412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.415696 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.415771 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.415774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.416667 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.416667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.417236 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.417238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.417675 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.417676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.417754 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.418415 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.418420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.418925 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.421132 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.421133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.421133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.421396 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.421644 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.421645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.421645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.421707 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.422148 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.422149 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.422194 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.422710 (+ 516us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.422934 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.254@KRBTEST.COM for oryx/127.16.99.253@KRBTEST.COM
I20260812 00:10:11.434698 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.425152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.16.99.253:35763)
0812 00:10:11.425196 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.425198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.425396 (+ 198us) server_negotiation.cc:407] Connection header received
0812 00:10:11.425471 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.425472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.425483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.425537 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.426169 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.426794 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.428131 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.428330 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.428391 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.431346 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.431347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.431347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.431357 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.433008 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.433631 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.433632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.433632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.433701 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.433979 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.433980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.433980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.434191 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.434262 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.434537 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.434619 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:10:11.434787 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.425056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.0.0.1:58710)
0812 00:10:11.425229 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.425290 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.425383 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.425551 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.425555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.425977 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.426076 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.426080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.426892 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.426893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.428025 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.428028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.428420 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.428421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.428532 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.429113 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.429120 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.431212 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.433157 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.433169 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.433172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.433511 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.433790 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.433791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.433792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.433880 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.434255 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.434266 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.434384 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.434499 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.434666 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260812 00:10:11.435905 23322 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.99.253:35763: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25"
member_type: NON_VOTER
last_known_addr {
host: "127.16.99.253"
port: 35763
}
attrs {
promote: true
}
}
dest_uuid: "d7225955d25c46e8a0d7b98372f77300"
cas_config_opid_index: -1
I20260812 00:10:11.436408 23322 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } }
I20260812 00:10:11.436972 23912 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9b3c555aad11484eb965ff159e1b9f25 (127.16.99.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } }
I20260812 00:10:11.437331 23855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:11.438053 23912 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.99.253:35763
I20260812 00:10:11.438445 23855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Refusing update from remote peer d7225955d25c46e8a0d7b98372f77300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 00:10:11.438482 23809 master.cc:537] Master@127.16.99.253:35763 shutting down...
I20260812 00:10:11.438823 23912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 00:10:11.439862 23305 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 00:10:11.440421 23305 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 -> Peer 9b3c555aad11484eb965ff159e1b9f25 (127.16.99.253:35763): Couldn't send request to peer 9b3c555aad11484eb965ff159e1b9f25. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 00:10:11.441169 23911 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } }
I20260812 00:10:11.441308 23911 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:11.442763 23809 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:11.442890 23809 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:11.442977 23809 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: stopping tablet replica
I20260812 00:10:11.456938 23809 master.cc:552] Master@127.16.99.253:35763 shutdown complete.
I20260812 00:10:11.457024 23809 master_runner.cc:305] Clearing existing system tablet
I20260812 00:10:11.457182 23809 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.457252 23809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.457989 23809 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260812 00:10:11.458585 23916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.458727 23809 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.458796 23809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.458847 23809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
uuid: "9b3c555aad11484eb965ff159e1b9f25"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "8a0e45062e8d8af4677ab72a6d2ad05d"
server_key_iv: "cbf5de202cf4c5d8f1789e677d305762"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.460096 23809 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:11.461794 23809 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 00:10:11.461896 23809 log.cc:1196] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 00:10:11.462116 23809 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Deleting consensus metadata
I20260812 00:10:11.462280 23809 master_runner.cc:315] Copying system tablet from 127.16.99.254:40411
I20260812 00:10:11.463064 23809 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: tablet copy: Beginning tablet copy session from remote peer at address 127.16.99.254:40411
I20260812 00:10:11.470660 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.463448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:59938)
0812 00:10:11.463562 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.463573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.463650 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.463811 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.463812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.463992 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.464052 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.464055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.465047 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.465049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.465655 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.465659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.466180 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.466181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.466333 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.467022 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.467027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.467400 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.469216 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.469217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.469217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.469422 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.469742 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.469743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.469744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.469823 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.470209 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.470210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.470260 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.470437 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.470558 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20260812 00:10:11.470659 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.463514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.16.99.254:40411)
0812 00:10:11.463574 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.463576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.463643 (+ 67us) server_negotiation.cc:407] Connection header received
0812 00:10:11.463708 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.463709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.463722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.463809 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.464130 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.464939 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.465753 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.466061 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.466147 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.467502 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.467504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.467504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.467513 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.469108 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.469507 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.469509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.469509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.469577 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.469909 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.469910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.469910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.470068 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.470147 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.470468 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.470558 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 00:10:11.471180 23343 tablet_copy_service.cc:140] P d7225955d25c46e8a0d7b98372f77300: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9b3c555aad11484eb965ff159e1b9f25 ({username='oryx', principal='oryx/127.16.99.253@KRBTEST.COM'} at 127.0.0.1:59938)
I20260812 00:10:11.471333 23343 tablet_copy_service.cc:161] P d7225955d25c46e8a0d7b98372f77300: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9b3c555aad11484eb965ff159e1b9f25 at {username='oryx', principal='oryx/127.16.99.253@KRBTEST.COM'} at 127.0.0.1:59938: session id = 9b3c555aad11484eb965ff159e1b9f25-00000000000000000000000000000000
I20260812 00:10:11.472024 23343 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 00:10:11.472752 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 00:10:11.473975 23809 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: tablet copy: Starting download of 0 data blocks...
I20260812 00:10:11.474157 23809 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: tablet copy: Starting download of 1 WAL segments...
I20260812 00:10:11.475324 23809 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 00:10:11.476399 23343 tablet_copy_service.cc:342] P d7225955d25c46e8a0d7b98372f77300: Request end of tablet copy session 9b3c555aad11484eb965ff159e1b9f25-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.99.253@KRBTEST.COM'} at 127.0.0.1:59938
I20260812 00:10:11.476462 23343 tablet_copy_service.cc:433] P d7225955d25c46e8a0d7b98372f77300: ending tablet copy session 9b3c555aad11484eb965ff159e1b9f25-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9b3c555aad11484eb965ff159e1b9f25
I20260812 00:10:11.479502 23809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:11.480942 23926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.481050 23928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.481144 23925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.481518 23809 server_base.cc:1034] running on GCE node
I20260812 00:10:11.481637 23809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:11.481878 23809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:11.483022 23809 hybrid_clock.cc:648] HybridClock initialized: now 1786493411483015 us; error 26 us; skew 500 ppm
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:11.484633 23809 init.cc:377] Logged in from keytab as oryx/127.16.99.253@KRBTEST.COM (short username oryx)
I20260812 00:10:11.484934 23809 webserver.cc:460] Webserver started at http://127.16.99.253:40269/ using document root <none> and password file <none>
I20260812 00:10:11.485059 23809 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.485095 23809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.485917 23809 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:11.486606 23935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.486774 23809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:11.486832 23809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
uuid: "9b3c555aad11484eb965ff159e1b9f25"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "8a0e45062e8d8af4677ab72a6d2ad05d"
server_key_iv: "cbf5de202cf4c5d8f1789e677d305762"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.486893 23809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.496695 23809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:11.498875 23809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:11.502306 23809 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.253:35763
I20260812 00:10:11.502837 23809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
I20260812 00:10:11.502862 23991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.253:35763 every 8 connection(s)
I20260812 00:10:11.503621 23992 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:10:11.504066 23992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Bootstrap starting.
I20260812 00:10:11.506496 23992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:10:11.506737 23992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: Bootstrap complete.
I20260812 00:10:11.506953 23992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } }
I20260812 00:10:11.507054 23992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b3c555aad11484eb965ff159e1b9f25, State: Initialized, Role: LEARNER
I20260812 00:10:11.507107 23992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } }
I20260812 00:10:11.507298 23992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:11.507326 23995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } }
I20260812 00:10:11.507431 23995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:10:11.508872 24006 catalog_manager.cc:1259] Loaded cluster ID: 6322e762f280494b82b3a368b222fb97
I20260812 00:10:11.508970 24006 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: loading cluster ID for follower catalog manager: success
I20260812 00:10:11.509918 24006 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: acquiring CA information for follower catalog manager: success
I20260812 00:10:11.510221 24006 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:10:11.557823 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.550027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.16.99.253:35763)
0812 00:10:11.550210 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.550213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.550226 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:11.550316 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.550317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.550329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.550441 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.551121 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.551943 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.552804 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.553057 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.553157 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.554699 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.554702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.554702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.554711 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.556426 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.556855 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.556857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.556858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.556928 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.557174 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.557175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.557176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.557313 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.557436 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.557577 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.557682 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.16.99.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.252.keytab.
Entry for principal oryx/127.16.99.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.252.keytab.
I20260812 00:10:11.593850 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:42653
--webserver_interface=127.16.99.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.252.keytab
--principal=oryx/127.16.99.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:11.682677 24010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:11.682943 24010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:11.682984 24010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:11.684316 24010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:11.684398 24010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:11.684423 24010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:11.684448 24010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:11.684494 24010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:11.686159 24010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.252.keytab
--principal=oryx/127.16.99.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:42653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.99.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:11.686502 24010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:11.686808 24010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:11.692385 24016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.692361 24015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.692346 24018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.692571 24010 server_base.cc:1034] running on GCE node
I20260812 00:10:11.692821 24010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:11.693149 24010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:11.694416 24010 hybrid_clock.cc:648] HybridClock initialized: now 1786493411694403 us; error 37 us; skew 500 ppm
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:11.697374 24010 init.cc:377] Logged in from keytab as oryx/127.16.99.252@KRBTEST.COM (short username oryx)
I20260812 00:10:11.697960 24010 webserver.cc:460] Webserver started at http://127.16.99.252:40887/ using document root <none> and password file <none>
I20260812 00:10:11.698207 24010 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.698287 24010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.698459 24010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:11.699584 24010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/instance:
uuid: "9be67bdc0b864d89b09f4e1450f2ada3"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "bffba02d0816d496a2eb82052b60d4f6"
server_key_iv: "4c2b8a23e5230bc96b49c764693719c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.699970 24010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal/instance:
uuid: "9be67bdc0b864d89b09f4e1450f2ada3"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "bffba02d0816d496a2eb82052b60d4f6"
server_key_iv: "4c2b8a23e5230bc96b49c764693719c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.701493 24010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:11.702453 24026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.702643 24010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:11.702746 24010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
uuid: "9be67bdc0b864d89b09f4e1450f2ada3"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "bffba02d0816d496a2eb82052b60d4f6"
server_key_iv: "4c2b8a23e5230bc96b49c764693719c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.702829 24010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.716856 24010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:11.719117 24010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:11.719352 24010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:11.723978 24010 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.252:42653
I20260812 00:10:11.724033 24080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.252:42653 every 8 connection(s)
I20260812 00:10:11.724426 24010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
I20260812 00:10:11.725479 24081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:11.727236 24081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:10:11.730438 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 24010
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.252@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.99.252@KRBTEST.COM
I20260812 00:10:11.737536 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.728055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.16.99.254:40411)
0812 00:10:11.728232 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.728234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.728245 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:11.728704 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.728706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.728717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.728784 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.729485 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.729994 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.731325 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.731540 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.731630 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.734334 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.734336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.734336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.734364 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.736010 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.736477 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.736479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.736479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.736539 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.736843 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.736844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.736845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.736966 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.737029 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.737361 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.737446 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260812 00:10:11.737738 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.727937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:59954)
0812 00:10:11.728109 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.728154 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.728601 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.728866 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.728869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.729133 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.729393 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.729398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.730101 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.730102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.731226 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.731228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.731666 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.731667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.731785 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.732305 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.732315 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.734231 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.736137 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.736146 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.736148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.736396 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.736616 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.736617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.736617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.736763 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.737044 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.737052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.737211 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.737321 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.737481 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260812 00:10:11.738402 24081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:10:11.741238 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.731052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.16.99.252:42653)
0812 00:10:11.731237 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.731241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.731258 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:10:11.731277 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.731279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.731295 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.731379 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.732026 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.733051 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.734113 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.734486 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.734579 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.737093 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.737101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.737102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.737107 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.739678 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.740196 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.740199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.740199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.740264 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.740470 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.740470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.740471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.740650 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.740858 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.741019 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.741102 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.252@KRBTEST.COM for oryx/127.16.99.253@KRBTEST.COM
I20260812 00:10:11.746716 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.738776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.16.99.253:35763)
0812 00:10:11.738845 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.738848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.738883 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:10:11.738952 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.738953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.738970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.739040 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.739290 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.740057 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.741312 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.741665 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.741815 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.743823 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.743824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.743824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.743832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.745318 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.745805 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.745806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.745806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.745871 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.746112 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.746113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.746113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.746239 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.746309 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.746562 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.746646 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:10:11.746717 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.738699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.0.0.1:58748)
0812 00:10:11.738837 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.738848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.738903 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.739040 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.739040 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.739150 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.739183 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.739184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.740146 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.740147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.741107 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.741110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.741781 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.741782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.741849 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.742295 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.742300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.743713 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.745421 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.745422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.745422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.745617 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.745959 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.745960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.745961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.746027 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.746411 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.746413 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.746472 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.746539 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.746644 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260812 00:10:11.747380 24081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.252@KRBTEST.COM for oryx/127.16.99.252@KRBTEST.COM
I20260812 00:10:11.755061 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.747678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.0.0.1:41012)
0812 00:10:11.747744 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.747757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.747807 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.747971 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.747972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.748187 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.748229 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.748231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.749101 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.749102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.749722 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.749725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.750255 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.750256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.750303 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.750800 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.750804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.752095 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.753811 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.753812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.753812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.753996 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.754257 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.754258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.754259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.754425 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.754733 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.754734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.754778 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.754845 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.754939 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:10:11.755074 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.747747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.16.99.252:42653)
0812 00:10:11.747834 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.747836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.747843 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:10:11.747859 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.747860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.747869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.747914 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.748375 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.749013 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.749822 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.750047 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.750112 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.752255 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.752257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.752257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.752264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.753719 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.754064 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.754065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.754065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.754113 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.754507 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.754508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.754509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.754639 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.754700 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.754881 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.754990 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":24}
I20260812 00:10:11.756855 24081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Bootstrap starting.
I20260812 00:10:11.757769 24081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:11.758025 24081 log.cc:824] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:11.758903 24081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: No bootstrap required, opened a new log
I20260812 00:10:11.760341 24081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } }
I20260812 00:10:11.760473 24081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:11.760495 24081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be67bdc0b864d89b09f4e1450f2ada3, State: Initialized, Role: FOLLOWER
I20260812 00:10:11.760627 24081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } }
I20260812 00:10:11.760926 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } } }
I20260812 00:10:11.760993 24081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:11.761018 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:10:11.762877 24097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:10:11.762941 24097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:10:11.766098 24010 master_runner.cc:428] Detected that this master 9be67bdc0b864d89b09f4e1450f2ada3 is joining an existing cluster
I20260812 00:10:11.766162 24010 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.99.252:42653
I20260812 00:10:11.774946 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.767519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.16.99.252:42653)
0812 00:10:11.767583 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.767586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.767616 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:10:11.767823 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.767824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.767834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.767884 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.768443 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.769208 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770392 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.770684 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770727 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.772006 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.772007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.772007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.772015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.773626 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.774021 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.774021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.774022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.774083 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.774285 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.774286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.774287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.774465 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.774544 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.774704 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.774803 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:10:11.775172 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.767320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.0.0.1:41020)
0812 00:10:11.767523 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.767559 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.767634 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.767927 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.767928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.768277 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.768357 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.768359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.769305 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.769306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.770251 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.770253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.770828 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.770895 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.771456 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.771461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.771874 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.773725 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.773726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.773727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.773906 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.774152 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.774153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.774153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.774212 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.774535 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.774535 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.774592 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.774682 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.774873 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":143,"thread_start_us":135,"threads_started":1}
I20260812 00:10:11.779453 24086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:11.779604 24086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } }
I20260812 00:10:11.779897 24086 leader_election.cc:288] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411), 9b3c555aad11484eb965ff159e1b9f25 (127.16.99.253:35763)
I20260812 00:10:11.780256 23333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7225955d25c46e8a0d7b98372f77300" is_pre_election: true
I20260812 00:10:11.780404 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3c555aad11484eb965ff159e1b9f25" is_pre_election: true
I20260812 00:10:11.780525 23963 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 LEARNER]: Handling vote request from an unknown peer 9be67bdc0b864d89b09f4e1450f2ada3
I20260812 00:10:11.780573 23963 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9be67bdc0b864d89b09f4e1450f2ada3 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260812 00:10:11.780834 24030 leader_election.cc:302] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9be67bdc0b864d89b09f4e1450f2ada3; no voters: 9b3c555aad11484eb965ff159e1b9f25, d7225955d25c46e8a0d7b98372f77300
I20260812 00:10:11.781037 24086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:11.782070 24086 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 00:10:11.794126 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 23387
I20260812 00:10:11.795120 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.767768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.16.99.254:40411)
0812 00:10:11.767827 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.767830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.767961 (+ 131us) server_negotiation.cc:407] Connection header received
0812 00:10:11.768062 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.768063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.768075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.768106 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.768583 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.769356 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770489 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.770789 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770833 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.791762 (+ 20929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.791764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.791764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.791775 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.793914 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.794379 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.794381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.794381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.794441 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.794620 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.794621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.794621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.794746 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.794786 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.794956 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.795037 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:10:11.795424 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.767712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:59966)
0812 00:10:11.767904 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.767913 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.767985 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.768171 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.768172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.768411 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.768481 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.768484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.769462 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.769463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.770382 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.770385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.770919 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.770986 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.791083 (+ 20097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.791090 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.791634 (+ 544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.794010 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.794012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.794012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.794256 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.794507 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.794508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.794508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.794560 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.794826 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.794827 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.794869 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.795204 (+ 335us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.795328 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":139,"threads_started":1}
I20260812 00:10:11.798647 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.767403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.16.99.253:35763)
0812 00:10:11.767583 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.767585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.767792 (+ 207us) server_negotiation.cc:407] Connection header received
0812 00:10:11.767897 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.767898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.767911 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.767942 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.768454 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.769208 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770383 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.770704 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770741 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.791711 (+ 20970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.791713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.791713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.791724 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.793963 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.797638 (+ 3675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.797640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.797640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.797714 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.797964 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.797966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.797966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.798137 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.798214 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.798456 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.798549 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260812 00:10:11.798664 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.767339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.0.0.1:58750)
0812 00:10:11.767737 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.767747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.767817 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.768009 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.768010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.768253 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.768357 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.768359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.769305 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.769306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.770251 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.770253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.770828 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.770829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.770895 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.791144 (+ 20249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.791149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.791555 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.797276 (+ 5721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.797277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.797278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.797537 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.797793 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.797794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.797794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.797874 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.798224 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.798225 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.798280 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.798417 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.798542 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":40,"thread_start_us":111,"threads_started":1}
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493410, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.254@KRBTEST.COM for oryx/127.16.99.252@KRBTEST.COM
I20260812 00:10:11.803606 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:40277
--local_ip_for_outbound_sockets=127.16.99.193
--tserver_master_addrs=127.16.99.254:40411
--webserver_port=33047
--webserver_interface=127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab
--principal=oryx/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:10:11.806996 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.799386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.0.0.1:41026)
0812 00:10:11.799443 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.799453 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.799491 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.799690 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.799692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.799869 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.799923 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.799925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.800533 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.800534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.801325 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.801328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.801696 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.801697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.801778 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.802177 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.802182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.803765 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.805714 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.805715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.805715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.805998 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.806224 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.806225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.806225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.806304 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.806653 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.806654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.806707 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.806771 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.806889 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:10:11.807402 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.799419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.16.99.252:42653)
0812 00:10:11.799562 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.799563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.799576 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:11.799601 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.799602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.799610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.799648 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.800000 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.800452 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.801426 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.801616 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.801679 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.803871 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.803873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.803873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.803881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.805613 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.806079 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.806081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.806081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.806146 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.806404 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.806405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.806405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.806551 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.806619 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.807210 (+ 591us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.807309 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260812 00:10:11.808122 23322 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.99.252:42653: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3"
member_type: NON_VOTER
last_known_addr {
host: "127.16.99.252"
port: 42653
}
attrs {
promote: true
}
}
dest_uuid: "d7225955d25c46e8a0d7b98372f77300"
cas_config_opid_index: 5
I20260812 00:10:11.808385 23322 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } }
I20260812 00:10:11.808753 23912 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.809737 23912 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.99.252:42653
I20260812 00:10:11.809818 23912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.809885 23912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:11.810410 24053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Refusing update from remote peer d7225955d25c46e8a0d7b98372f77300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 00:10:11.810745 24010 master.cc:537] Master@127.16.99.252:42653 shutting down...
I20260812 00:10:11.810750 23911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 00:10:11.811831 23306 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 -> Peer 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252:42653): Couldn't send request to peer 9be67bdc0b864d89b09f4e1450f2ada3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 00:10:11.814275 24010 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:11.814404 24010 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:11.814457 24010 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: stopping tablet replica
I20260812 00:10:11.815197 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.808941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.16.99.253:35763)
0812 00:10:11.809118 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.809120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.809271 (+ 151us) server_negotiation.cc:407] Connection header received
0812 00:10:11.809311 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.809312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.809341 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.809402 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:11.809601 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.810453 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.813506 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.814175 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.814253 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.814564 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.815051 (+ 487us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.815105 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260812 00:10:11.815526 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.808832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.0.0.1:58758)
0812 00:10:11.809154 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.809167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.809218 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.809468 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.809469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.809471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:11.809523 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.809524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.811999 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.812001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.813384 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.813388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.814604 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.814605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.814686 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.814717 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.814779 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.814827 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304}
I20260812 00:10:11.815953 23963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 LEARNER]: Refusing update from remote peer d7225955d25c46e8a0d7b98372f77300: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 00:10:11.816205 23911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 00:10:11.816874 23963 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.817647 24010 master.cc:552] Master@127.16.99.252:42653 shutdown complete.
I20260812 00:10:11.817685 24010 master_runner.cc:305] Clearing existing system tablet
I20260812 00:10:11.817823 24010 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.817864 24010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.818498 23995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7225955d25c46e8a0d7b98372f77300. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.818586 23995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:10:11.818622 23995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: NON_VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: true } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.818660 23995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:10:11.819504 23911 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: attempting to promote NON_VOTER 9b3c555aad11484eb965ff159e1b9f25 to VOTER
I20260812 00:10:11.819729 23911 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } }
I20260812 00:10:11.820226 23963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 LEARNER]: Refusing update from remote peer d7225955d25c46e8a0d7b98372f77300: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 00:10:11.820384 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.820170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.0.0.1:41042)
0812 00:10:11.820225 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.820266 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:42653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:10:11.820478 23911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 00:10:11.820657 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.820590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.0.0.1:41054)
0812 00:10:11.820620 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.820639 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:42653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260812 00:10:11.820708 23306 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 -> Peer 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252:42653): Couldn't send request to peer 9be67bdc0b864d89b09f4e1450f2ada3. Status: Network error: Client connection negotiation failed: client connection to 127.16.99.252:42653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 00:10:11.820984 24010 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260812 00:10:11.821691 24112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.821795 24010 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.821743 24111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9b3c555aad11484eb965ff159e1b9f25 (127.16.99.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.821834 24010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.821856 24010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
uuid: "9be67bdc0b864d89b09f4e1450f2ada3"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "bffba02d0816d496a2eb82052b60d4f6"
server_key_iv: "4c2b8a23e5230bc96b49c764693719c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.822724 24010 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:11.822799 24111 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.822872 24111 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:11.822921 24111 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.822962 24111 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:11.823429 23963 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9b3c555aad11484eb965ff159e1b9f25 (127.16.99.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.824446 24109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.824566 24109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:10:11.824909 24010 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 00:10:11.824990 24010 log.cc:1196] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 00:10:11.825202 24010 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Deleting consensus metadata
I20260812 00:10:11.825335 24010 master_runner.cc:315] Copying system tablet from 127.16.99.254:40411
I20260812 00:10:11.825985 24010 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: tablet copy: Beginning tablet copy session from remote peer at address 127.16.99.254:40411
I20260812 00:10:11.833863 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.826319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.16.99.254:40411)
0812 00:10:11.826417 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.826419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.826698 (+ 279us) server_negotiation.cc:407] Connection header received
0812 00:10:11.826752 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.826753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.826765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.826834 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.827226 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.828037 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.829151 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.829470 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.829543 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.830846 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.830848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.830849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.830856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.832556 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.832998 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.832999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.833000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.833063 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.833268 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.833269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.833270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.833437 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.833505 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.833685 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.833773 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 00:10:11.833891 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.826470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:59982)
0812 00:10:11.826608 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.826621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.826690 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.826849 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.826849 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.827066 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.827136 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.827138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.828134 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.828135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.829027 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.829030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.829556 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.829558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.829633 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.830171 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.830177 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.830746 (+ 569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.832659 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.832660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.832661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.832914 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.833135 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.833136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.833136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.833204 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.833512 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.833513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.833567 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.833644 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.833768 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260812 00:10:11.834663 23343 tablet_copy_service.cc:140] P d7225955d25c46e8a0d7b98372f77300: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9be67bdc0b864d89b09f4e1450f2ada3 ({username='oryx', principal='oryx/127.16.99.252@KRBTEST.COM'} at 127.0.0.1:59982)
I20260812 00:10:11.834728 23343 tablet_copy_service.cc:161] P d7225955d25c46e8a0d7b98372f77300: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9be67bdc0b864d89b09f4e1450f2ada3 at {username='oryx', principal='oryx/127.16.99.252@KRBTEST.COM'} at 127.0.0.1:59982: session id = 9be67bdc0b864d89b09f4e1450f2ada3-00000000000000000000000000000000
I20260812 00:10:11.835361 23343 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 00:10:11.836136 24010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 00:10:11.837630 24010 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: tablet copy: Starting download of 0 data blocks...
I20260812 00:10:11.837800 24010 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: tablet copy: Starting download of 1 WAL segments...
I20260812 00:10:11.838872 24010 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 00:10:11.839969 23343 tablet_copy_service.cc:342] P d7225955d25c46e8a0d7b98372f77300: Request end of tablet copy session 9be67bdc0b864d89b09f4e1450f2ada3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.99.252@KRBTEST.COM'} at 127.0.0.1:59982
I20260812 00:10:11.840027 23343 tablet_copy_service.cc:433] P d7225955d25c46e8a0d7b98372f77300: ending tablet copy session 9be67bdc0b864d89b09f4e1450f2ada3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9be67bdc0b864d89b09f4e1450f2ada3
I20260812 00:10:11.843017 24010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:11.844542 24123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.844653 24010 server_base.cc:1034] running on GCE node
W20260812 00:10:11.844550 24120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.846516 24121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.846716 24010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:11.846873 24010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:11.847972 24010 hybrid_clock.cc:648] HybridClock initialized: now 1786493411847951 us; error 40 us; skew 500 ppm
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:11.849645 24010 init.cc:377] Logged in from keytab as oryx/127.16.99.252@KRBTEST.COM (short username oryx)
I20260812 00:10:11.849951 24010 webserver.cc:460] Webserver started at http://127.16.99.252:45279/ using document root <none> and password file <none>
I20260812 00:10:11.850077 24010 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.850116 24010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.850946 24010 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:10:11.851692 24130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.851869 24010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:10:11.851928 24010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
uuid: "9be67bdc0b864d89b09f4e1450f2ada3"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "bffba02d0816d496a2eb82052b60d4f6"
server_key_iv: "4c2b8a23e5230bc96b49c764693719c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.851998 24010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.876673 24010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:11.878759 24010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:11.882171 24010 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.252:42653
I20260812 00:10:11.882721 24010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
I20260812 00:10:11.882982 24182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.252:42653 every 8 connection(s)
I20260812 00:10:11.883849 24183 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:10:11.884400 24183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Bootstrap starting.
I20260812 00:10:11.887668 24183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:10:11.887961 24183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Bootstrap complete.
I20260812 00:10:11.888199 24183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } }
I20260812 00:10:11.888304 24183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9be67bdc0b864d89b09f4e1450f2ada3, State: Initialized, Role: LEARNER
I20260812 00:10:11.888345 24183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } }
I20260812 00:10:11.888567 24183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:11.890271 24198 catalog_manager.cc:1259] Loaded cluster ID: 6322e762f280494b82b3a368b222fb97
I20260812 00:10:11.890453 24198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: loading cluster ID for follower catalog manager: success
I20260812 00:10:11.890618 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:11.891481 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:10:11.892666 24198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: acquiring CA information for follower catalog manager: success
I20260812 00:10:11.893077 24198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 00:10:11.914739 24107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:11.914973 24107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:11.915001 24107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:11.916270 24107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:11.916318 24107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:11.916348 24107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:11.918023 24107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab
--principal=oryx/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:40277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=33047
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:11.918311 24107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:11.918596 24107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:11.924599 24203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:11.924618 24206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.924894 24107 server_base.cc:1034] running on GCE node
W20260812 00:10:11.924638 24204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:11.925326 24107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:11.925722 24107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:11.926930 24107 hybrid_clock.cc:648] HybridClock initialized: now 1786493411926872 us; error 77 us; skew 500 ppm
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:11.929926 24107 init.cc:377] Logged in from keytab as oryx/127.16.99.193@KRBTEST.COM (short username oryx)
I20260812 00:10:11.930603 24107 webserver.cc:460] Webserver started at http://127.16.99.193:33047/ using document root <none> and password file <none>
I20260812 00:10:11.930907 24107 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:11.930994 24107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:11.932502 24107 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:10:11.933240 24215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.933458 24107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:11.933542 24107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "ffa32349f97846248d6f3b2faa2ed898"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "18870cead4c946ba649ec1c8e5f6546f"
server_key_iv: "6cfdc063b86b692c817bf8a2dcd913d9"
server_key_version: "encryptionkey@0"
I20260812 00:10:11.933990 24107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:11.946914 24107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:11.949632 24107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:11.949803 24107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:11.950057 24107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:11.950519 24107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:11.950577 24107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.950603 24107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:11.950655 24107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:11.957703 24107 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:40277
I20260812 00:10:11.957747 24334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:40277 every 8 connection(s)
I20260812 00:10:11.958184 24107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:11.961248 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 24107
I20260812 00:10:11.961439 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 23525
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.253@KRBTEST.COM
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:11.971861 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.959627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:55691 (local address 127.16.99.254:40411)
0812 00:10:11.959703 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.959705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.960604 (+ 899us) server_negotiation.cc:407] Connection header received
0812 00:10:11.960642 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.960643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.960656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.960709 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.962107 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.962970 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.964596 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.965008 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.965057 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.968556 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.968558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.968558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.968567 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.970044 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.970754 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.970756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.970756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.970840 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.971110 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.971111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.971112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.971273 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.971346 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.971663 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.971752 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:10:11.971926 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.959151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:37077 (local address 127.16.99.253:35763)
0812 00:10:11.959247 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.959250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.959462 (+ 212us) server_negotiation.cc:407] Connection header received
0812 00:10:11.960575 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.960577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.960591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.960654 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.962074 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.962848 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.964427 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.964705 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.964742 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.968229 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.968231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.968231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.968239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.970153 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.970937 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.970938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.970938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.970997 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.971184 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.971185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.971186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.971352 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.971420 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.971680 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.971775 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 00:10:11.972476 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.959059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.16.99.193:37077)
0812 00:10:11.959324 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.959371 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.960445 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.960785 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.960789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.961471 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.961981 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.961983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.962997 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.962999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.964326 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.964329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.964828 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.964829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.964974 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.965532 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.965540 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.968097 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.970226 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.970227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.970227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.970669 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.971065 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.971066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.971067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.971127 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.971416 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.971417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.971536 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.972012 (+ 476us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.972240 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":160,"thread_start_us":131,"threads_started":1}
I20260812 00:10:11.972560 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:44107
--local_ip_for_outbound_sockets=127.16.99.194
--tserver_master_addrs=127.16.99.254:40411
--webserver_port=46841
--webserver_interface=127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab
--principal=oryx/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:10:11.972474 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.959399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.193:55691)
0812 00:10:11.960556 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.960563 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.960602 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.960780 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.960789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.961439 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.961991 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.961994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.963060 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.963061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.964472 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.964475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.965150 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.965151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.965222 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.965681 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.965687 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.968463 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.970155 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.970167 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.970170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.970669 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.970916 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.970917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.970917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.971047 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.971346 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.971354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.971527 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.971639 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.972156 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":488,"thread_start_us":77,"threads_started":1}
I20260812 00:10:11.973340 24336 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40411
I20260812 00:10:11.973441 24336 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:11.973641 24336 heartbeater.cc:511] Master 127.16.99.254:40411 requested a full tablet report, sending...
I20260812 00:10:11.973809 24337 heartbeater.cc:347] Connected to a master server at 127.16.99.253:35763
I20260812 00:10:11.973855 24337 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:11.973915 24337 heartbeater.cc:511] Master 127.16.99.253:35763 requested a full tablet report, sending...
I20260812 00:10:11.974047 23322 ts_manager.cc:194] Re-registered known tserver with Master: ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
I20260812 00:10:11.974309 23952 ts_manager.cc:194] Registered new tserver with Master: ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
I20260812 00:10:11.974586 23322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:55691
Aug 12 00:10:11 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493411, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.252@KRBTEST.COM
I20260812 00:10:11.979755 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.959337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:44275 (local address 127.16.99.252:42653)
0812 00:10:11.960720 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:11.960722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:11.960730 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:11.960755 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:11.960755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:11.960767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:11.960834 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:11.972153 (+ 11319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.973094 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.974104 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.974412 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.974805 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.976501 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:11.976503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:11.976503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:11.976512 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:11.978397 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.978800 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.978802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.978802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.978874 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:11.979073 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:11.979074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:11.979074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:11.979238 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:11.979308 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:11.979498 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 00:10:11.979607 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":62,"threads_started":1}
I20260812 00:10:11.979804 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:11.959058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.16.99.193:44275)
0812 00:10:11.959324 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:11.959371 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:11.960412 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:11.960861 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:11.960862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:11.961448 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:11.961832 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.961838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.973178 (+ 11340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.973179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.974002 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:11.974005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:11.974524 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:11.974525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:11.974597 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:11.975117 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:11.975121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:11.976410 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:11.978512 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.978514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.978514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.978709 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.978955 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:11.978956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:11.978956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:11.979010 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:11.979344 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:11.979345 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:11.979386 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 00:10:11.979444 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 00:10:11.979542 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":16,"thread_start_us":111,"threads_started":1}
I20260812 00:10:11.980383 24335 heartbeater.cc:347] Connected to a master server at 127.16.99.252:42653
I20260812 00:10:11.980456 24335 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:11.980566 24335 heartbeater.cc:511] Master 127.16.99.252:42653 requested a full tablet report, sending...
I20260812 00:10:11.981006 24147 ts_manager.cc:194] Registered new tserver with Master: ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
W20260812 00:10:12.065692 24343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:12.065953 24343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:12.065985 24343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:12.067497 24343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:12.067562 24343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:12.067597 24343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:12.069401 24343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab
--principal=oryx/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:44107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=46841
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:12.069702 24343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:12.070008 24343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:12.075726 24348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:12.075726 24351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:12.075784 24343 server_base.cc:1034] running on GCE node
W20260812 00:10:12.075719 24349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:12.076372 24343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:12.076618 24343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:12.077814 24343 hybrid_clock.cc:648] HybridClock initialized: now 1786493412077809 us; error 43 us; skew 500 ppm
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:12.080700 24343 init.cc:377] Logged in from keytab as oryx/127.16.99.194@KRBTEST.COM (short username oryx)
I20260812 00:10:12.081152 24343 webserver.cc:460] Webserver started at http://127.16.99.194:46841/ using document root <none> and password file <none>
I20260812 00:10:12.081349 24343 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:12.081408 24343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:12.082917 24343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:12.083628 24360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:12.083899 24343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:12.083997 24343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "b376fa291da84a088858823be12e036d"
format_stamp: "Formatted at 2026-08-12 00:10:10 on dist-test-slave-zpfg"
server_key: "851387cba72752123aa537d01db8f189"
server_key_iv: "fa8f9fba7850bc6cc1457965d16862c0"
server_key_version: "encryptionkey@0"
I20260812 00:10:12.084385 24343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:12.100879 24343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:12.103600 24343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:12.103724 24343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:12.103950 24343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:12.104223 24343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:12.104257 24343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:12.104281 24343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:12.104295 24343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:12.110767 24343 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:44107
I20260812 00:10:12.110816 24476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:44107 every 8 connection(s)
I20260812 00:10:12.111193 24343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.252@KRBTEST.COM
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.253@KRBTEST.COM
I20260812 00:10:12.120105 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 24343
I20260812 00:10:12.120281 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 23666
I20260812 00:10:12.125270 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.112114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:49335 (local address 127.16.99.252:42653)
0812 00:10:12.112209 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.112211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.113958 (+ 1747us) server_negotiation.cc:407] Connection header received
0812 00:10:12.113995 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.113996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.114012 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.114074 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.114678 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.115450 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.116646 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.116947 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.117027 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.119861 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.119863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.119863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.119872 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.123272 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.124057 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.124058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.124059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.124124 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.124474 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.124476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.124476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.124672 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.124763 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.125068 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.125172 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 00:10:12.125582 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.112061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:54903 (local address 127.16.99.253:35763)
0812 00:10:12.112169 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.112172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.112567 (+ 395us) server_negotiation.cc:407] Connection header received
0812 00:10:12.113533 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.113534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.113544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.113604 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.114677 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.115445 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.117090 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.117333 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.117413 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.120202 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.120204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.120204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.120213 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.123635 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.124102 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.124103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.124104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.124183 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.124811 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.124812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.124813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.124995 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.125073 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.125318 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.125426 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 00:10:12.125854 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.111937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.16.99.194:49335)
0812 00:10:12.113877 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.113887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.113944 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.114103 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.114104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.114301 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:12.114556 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.114562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.115549 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.115550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.116552 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.116555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.117088 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.117089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.117164 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.117732 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:12.117736 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:12.119762 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:12.123378 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.123392 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.123395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.123954 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.124205 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.124205 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.124206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.124390 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.124755 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:12.124762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:12.124915 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.125043 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.125234 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1913,"mutex_wait_us":207,"thread_start_us":83,"threads_started":1}
I20260812 00:10:12.126055 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.111937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.16.99.194:54903)
0812 00:10:12.112422 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.112471 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.113426 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.113728 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.113733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.114282 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:12.114556 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.114562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.115549 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.115550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.116552 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.116555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.117442 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.117443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.117538 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.118268 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:12.118272 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:12.120088 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:12.123720 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.123722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.123722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.123973 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.124248 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.124248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.124248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.124390 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.125108 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:12.125109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:12.125162 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.125698 (+ 536us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.125852 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":86,"thread_start_us":89,"threads_started":1}
I20260812 00:10:12.126616 24479 heartbeater.cc:347] Connected to a master server at 127.16.99.253:35763
I20260812 00:10:12.126736 24479 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:12.127009 24479 heartbeater.cc:511] Master 127.16.99.253:35763 requested a full tablet report, sending...
I20260812 00:10:12.127545 24477 heartbeater.cc:347] Connected to a master server at 127.16.99.252:42653
I20260812 00:10:12.127599 23952 ts_manager.cc:194] Registered new tserver with Master: b376fa291da84a088858823be12e036d (127.16.99.194:44107)
I20260812 00:10:12.127642 24477 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:12.127769 24477 heartbeater.cc:511] Master 127.16.99.252:42653 requested a full tablet report, sending...
I20260812 00:10:12.128046 24147 ts_manager.cc:194] Registered new tserver with Master: b376fa291da84a088858823be12e036d (127.16.99.194:44107)
I20260812 00:10:12.135198 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:44283
--local_ip_for_outbound_sockets=127.16.99.195
--tserver_master_addrs=127.16.99.254:40411
--webserver_port=34801
--webserver_interface=127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab
--principal=oryx/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:10:12.137475 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.112252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:38217 (local address 127.16.99.254:40411)
0812 00:10:12.112362 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.112364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.113683 (+ 1319us) server_negotiation.cc:407] Connection header received
0812 00:10:12.113721 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.113722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.113733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.113802 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.114650 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.115171 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.116331 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.116536 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.116570 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.119191 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.119192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.119193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.119201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.135956 (+ 16755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.136455 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.136457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.136457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.136538 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.136739 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.136740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.136741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.136916 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.136996 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.137256 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.137407 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 00:10:12.137892 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.111937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.194:38217)
0812 00:10:12.113578 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.113617 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.113670 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.113837 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.113838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.114248 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:12.114555 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.114560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.115282 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.115283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.116167 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.116169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.116682 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.116683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.116829 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.117460 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:12.117477 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:12.119059 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:12.136102 (+ 17043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.136104 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.136104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.136365 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.136614 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.136615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.136616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.136676 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.137048 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:12.137049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:12.137113 (+ 64us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.137215 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.137365 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":9,"thread_start_us":96,"threads_started":1}
I20260812 00:10:12.138496 24478 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40411
I20260812 00:10:12.138571 24478 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:12.138686 24478 heartbeater.cc:511] Master 127.16.99.254:40411 requested a full tablet report, sending...
I20260812 00:10:12.138988 23322 ts_manager.cc:194] Re-registered known tserver with Master: b376fa291da84a088858823be12e036d (127.16.99.194:44107)
I20260812 00:10:12.139394 23322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:38217
W20260812 00:10:12.227659 24484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:12.227903 24484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:12.227932 24484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:12.229287 24484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:12.229349 24484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:12.229380 24484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:12.231132 24484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab
--principal=oryx/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:44283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=34801
--tserver_master_addrs=127.16.99.254:40411,127.16.99.253:35763,127.16.99.252:42653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:12.231419 24484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:12.231702 24484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:12.237633 24492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:12.237690 24489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:12.237640 24490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:12.238117 24484 server_base.cc:1034] running on GCE node
I20260812 00:10:12.238390 24484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:12.238694 24484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:12.239882 24484 hybrid_clock.cc:648] HybridClock initialized: now 1786493412239882 us; error 65 us; skew 500 ppm
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:12.249150 24484 init.cc:377] Logged in from keytab as oryx/127.16.99.195@KRBTEST.COM (short username oryx)
I20260812 00:10:12.249758 24484 webserver.cc:460] Webserver started at http://127.16.99.195:34801/ using document root <none> and password file <none>
I20260812 00:10:12.250041 24484 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:12.250144 24484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:12.251959 24484 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:12.252710 24499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:12.253050 24484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:12.253185 24484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "ae1a1ad80ebf4d6fa74b3c6020662589"
format_stamp: "Formatted at 2026-08-12 00:10:11 on dist-test-slave-zpfg"
server_key: "cd9ddfdd1f8054f5177e783fc8b50698"
server_key_iv: "92172d394c43d02ef49c49b41cd732fc"
server_key_version: "encryptionkey@0"
I20260812 00:10:12.253608 24484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:12.260772 24484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:12.263381 24484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:12.263563 24484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:12.263849 24484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:12.264189 24484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:12.264243 24484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:12.264288 24484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:12.264322 24484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:12.271667 24484 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:44283
I20260812 00:10:12.271735 24618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:44283 every 8 connection(s)
I20260812 00:10:12.272213 24484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:12.272384 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 24484
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.252@KRBTEST.COM
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.253@KRBTEST.COM
Aug 12 00:10:12 dist-test-slave-zpfg krb5kdc[23275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493412, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:12.286468 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.273084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.16.99.195:33485)
0812 00:10:12.273272 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.273323 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.274149 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.274659 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.274660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.275164 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:12.275505 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.275512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.276706 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.276707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.278172 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.278174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.278771 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.278923 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.279830 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:12.279842 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:12.281953 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:12.284255 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.284269 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.284272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.284660 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.284924 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.284925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.284926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.285102 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.285492 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:12.285513 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:12.285720 (+ 207us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.285859 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.286079 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":82,"thread_start_us":81,"threads_started":1}
I20260812 00:10:12.287011 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.273583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:33485 (local address 127.16.99.252:42653)
0812 00:10:12.273630 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.273632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.273653 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:10:12.274566 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.274567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.274578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.274648 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.275627 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.276569 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278296 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.278642 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278709 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.282079 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.282081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.282081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.282090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.284134 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.284760 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.284762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.284763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.284834 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.285196 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.285197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.285197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.285381 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.285457 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.286787 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.286896 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:10:12.287436 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.273752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58151 (local address 127.16.99.253:35763)
0812 00:10:12.273827 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.273829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.275227 (+ 1398us) server_negotiation.cc:407] Connection header received
0812 00:10:12.275288 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.275289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.275302 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.275369 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.276276 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.277068 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.279087 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.279423 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.279475 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.283421 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.283422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.283423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.283430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.285189 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.286046 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.286046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.286047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.286093 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.286758 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.286760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.286760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.286903 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.286954 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.287209 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.287311 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:10:12.287928 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.273402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.16.99.195:58151)
0812 00:10:12.275138 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.275167 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.275232 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.275421 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.275422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.275622 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:12.275683 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.275685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.277175 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.277176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.278172 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.278174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.279556 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.279557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.279623 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.280054 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:12.280058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:12.283018 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:12.285266 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.285267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.285267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.285477 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.286177 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.286178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.286179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.286673 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.286988 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:12.286989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:12.287082 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.287592 (+ 510us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.287757 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1718,"mutex_wait_us":11,"thread_start_us":66,"threads_started":1}
I20260812 00:10:12.288081 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.274281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.16.99.253:35763)
0812 00:10:12.274575 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.274577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.275701 (+ 1124us) server_negotiation.cc:407] Connection header received
0812 00:10:12.275728 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.275729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.275742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.275797 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.276522 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.277260 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278275 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.278605 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278668 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.280505 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.280506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.280507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.280513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.284403 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.285839 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.285841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.285841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.285910 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.286247 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.286248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.286248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.286416 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.286545 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.287889 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.287981 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":80,"threads_started":1}
I20260812 00:10:12.288311 24619 heartbeater.cc:347] Connected to a master server at 127.16.99.252:42653
I20260812 00:10:12.288424 24619 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:12.288592 24621 heartbeater.cc:347] Connected to a master server at 127.16.99.253:35763
I20260812 00:10:12.288666 24619 heartbeater.cc:511] Master 127.16.99.252:42653 requested a full tablet report, sending...
I20260812 00:10:12.288666 24621 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:12.288861 24621 heartbeater.cc:511] Master 127.16.99.253:35763 requested a full tablet report, sending...
I20260812 00:10:12.289363 23952 ts_manager.cc:194] Registered new tserver with Master: ae1a1ad80ebf4d6fa74b3c6020662589 (127.16.99.195:44283)
I20260812 00:10:12.289403 24147 ts_manager.cc:194] Registered new tserver with Master: ae1a1ad80ebf4d6fa74b3c6020662589 (127.16.99.195:44283)
I20260812 00:10:12.290921 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.273430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:44675 (local address 127.16.99.254:40411)
0812 00:10:12.273554 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.273556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.274416 (+ 860us) server_negotiation.cc:407] Connection header received
0812 00:10:12.274455 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.274456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.274470 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.274516 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.275729 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.276571 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278287 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.278614 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278668 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.283764 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.283765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.283766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.283774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.286239 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.289891 (+ 3652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.289892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.289893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.289964 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.290233 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.290234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.290234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.290448 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.290542 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.290736 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.290823 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 00:10:12.291399 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.275479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.16.99.254:40411)
0812 00:10:12.279249 (+ 3770us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.279252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.280133 (+ 881us) server_negotiation.cc:407] Connection header received
0812 00:10:12.280179 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.280180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.280198 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.280273 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.282438 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.283233 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.285157 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.285483 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.285544 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.287138 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.287152 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.287152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.287160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.289789 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.290266 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.290268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.290269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.290364 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.290585 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.290586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.290587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.290733 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.290833 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.291194 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.291282 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3718,"thread_start_us":82,"threads_started":1}
I20260812 00:10:12.291410 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.273138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.195:44675)
0812 00:10:12.274297 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.274309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.274400 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.274605 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.274611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.275165 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:12.275616 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.275619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.276715 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.276715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.278171 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.278174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.278766 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.278766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.278930 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.279829 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:12.279840 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:12.283312 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:12.286369 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.286370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.286371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.288882 (+ 2511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.290057 (+ 1175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:12.290058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:12.290059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:12.290134 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:12.290534 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:12.290535 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:12.290595 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.290686 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.290818 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":50,"thread_start_us":78,"threads_started":1}
I20260812 00:10:12.291755 24620 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40411
I20260812 00:10:12.291810 24620 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:12.291893 24620 heartbeater.cc:511] Master 127.16.99.254:40411 requested a full tablet report, sending...
I20260812 00:10:12.292150 23322 ts_manager.cc:194] Re-registered known tserver with Master: ae1a1ad80ebf4d6fa74b3c6020662589 (127.16.99.195:44283)
I20260812 00:10:12.292594 23322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:44675
I20260812 00:10:12.293094 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.273915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.16.99.252:42653)
0812 00:10:12.274047 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.274049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.274372 (+ 323us) server_negotiation.cc:407] Connection header received
0812 00:10:12.274527 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.274529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.274546 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.274592 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.278296 (+ 3704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.280105 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.281099 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.281347 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.281439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.289598 (+ 8159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.289600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.289600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.289609 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.291518 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.292024 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.292025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.292026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.292095 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.292335 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.292337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.292337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.292514 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.292616 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.292866 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.292977 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260812 00:10:12.293642 23322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:12.295393 23322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:12.304536 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.299616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44107 (local address 127.0.0.1:60692)
0812 00:10:12.299729 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.299743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.299793 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.300207 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.300208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.300210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:12.300262 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.300265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.301969 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.301970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.303401 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.303405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.304276 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.304277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.304341 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.304345 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.304396 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.304445 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 00:10:12.304548 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.299738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.16.99.194:44107)
0812 00:10:12.299982 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.299989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.300001 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:12.300050 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.300053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.300074 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.300215 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:12.300359 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.301863 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.303522 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.304168 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.304289 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.304390 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.304419 (+ 29us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.304462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":132,"threads_started":1}
I20260812 00:10:12.304979 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.300083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:40277 (local address 127.0.0.1:49844)
0812 00:10:12.300358 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.300368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.300410 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.300819 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.300820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.300822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:12.300862 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.300864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.302189 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.302190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.303529 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.303531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.304681 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.304682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.304773 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.304779 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.304835 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.304886 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":211,"threads_started":1}
I20260812 00:10:12.305320 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.300451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.16.99.193:40277)
0812 00:10:12.300616 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.300622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.300634 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:12.300679 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.300681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.300698 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.300791 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:12.300925 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.302071 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.303633 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.304474 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.304553 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.304640 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.305176 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.305231 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260812 00:10:12.305438 24410 tablet_service.cc:1467] Processing CreateTablet for tablet 32f055da851846e380a8384b8dd66bcc (DEFAULT_TABLE table=test-table [id=975247fa01c3457ab0f51a77b7730673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:12.305897 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f055da851846e380a8384b8dd66bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:12.306046 24267 tablet_service.cc:1467] Processing CreateTablet for tablet 32f055da851846e380a8384b8dd66bcc (DEFAULT_TABLE table=test-table [id=975247fa01c3457ab0f51a77b7730673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:12.306452 24267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f055da851846e380a8384b8dd66bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:12.306854 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.299954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.16.99.195:44283)
0812 00:10:12.302028 (+ 2074us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.302034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.302045 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:12.302090 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.302092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.302111 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.302196 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:12.302517 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.303352 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.305472 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.306304 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.306434 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.306525 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.306701 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.306754 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2024,"thread_start_us":88,"threads_started":1}
I20260812 00:10:12.306857 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.299847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:44283 (local address 127.0.0.1:33616)
0812 00:10:12.300020 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.300032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.300089 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.302320 (+ 2231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.302321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.302324 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:12.302418 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.302419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.303458 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.303459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.305360 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.305363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.306458 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.306460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.306590 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.306595 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.306668 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.306766 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":121,"threads_started":1}
I20260812 00:10:12.307978 24549 tablet_service.cc:1467] Processing CreateTablet for tablet 32f055da851846e380a8384b8dd66bcc (DEFAULT_TABLE table=test-table [id=975247fa01c3457ab0f51a77b7730673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:12.308418 24549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f055da851846e380a8384b8dd66bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:12.310034 24646 tablet_bootstrap.cc:492] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Bootstrap starting.
I20260812 00:10:12.310485 24647 tablet_bootstrap.cc:492] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Bootstrap starting.
I20260812 00:10:12.310869 24646 tablet_bootstrap.cc:654] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:12.311254 24647 tablet_bootstrap.cc:654] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:12.311264 24646 log.cc:824] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:12.311635 24647 log.cc:824] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:12.312389 24650 tablet_bootstrap.cc:492] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Bootstrap starting.
I20260812 00:10:12.312425 24647 tablet_bootstrap.cc:492] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: No bootstrap required, opened a new log
I20260812 00:10:12.312498 24647 ts_tablet_manager.cc:1397] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:10:12.313086 24646 tablet_bootstrap.cc:492] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: No bootstrap required, opened a new log
I20260812 00:10:12.313120 24650 tablet_bootstrap.cc:654] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:12.313171 24646 ts_tablet_manager.cc:1397] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 00:10:12.313503 24650 log.cc:824] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:12.314260 24650 tablet_bootstrap.cc:492] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: No bootstrap required, opened a new log
I20260812 00:10:12.314348 24650 ts_tablet_manager.cc:1397] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:10:12.315449 24647 raft_consensus.cc:348] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.315662 24647 raft_consensus.cc:374] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:12.315699 24647 raft_consensus.cc:729] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa32349f97846248d6f3b2faa2ed898, State: Initialized, Role: FOLLOWER
I20260812 00:10:12.315847 24647 consensus_queue.cc:260] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.316170 24647 ts_tablet_manager.cc:1428] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 00:10:12.316031 24646 raft_consensus.cc:348] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.316213 24646 raft_consensus.cc:374] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:12.316294 24336 heartbeater.cc:503] Master 127.16.99.254:40411 was elected leader, sending a full tablet report...
I20260812 00:10:12.316251 24646 raft_consensus.cc:729] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b376fa291da84a088858823be12e036d, State: Initialized, Role: FOLLOWER
I20260812 00:10:12.316804 24646 consensus_queue.cc:260] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.317096 24646 ts_tablet_manager.cc:1428] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260812 00:10:12.317530 24478 heartbeater.cc:503] Master 127.16.99.254:40411 was elected leader, sending a full tablet report...
I20260812 00:10:12.319372 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.315011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.0.0.1:41066)
0812 00:10:12.315048 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.315063 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.315109 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.315472 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.315474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.315476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:12.315525 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.315528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.316832 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.316833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.317984 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.317987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.319087 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.319088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.319165 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.319171 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.319236 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.319286 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 00:10:12.319636 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.315203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.16.99.252:42653)
0812 00:10:12.315272 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.315274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.315285 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:12.315316 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.315317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.315346 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.315414 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:12.315617 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.316500 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.318219 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.318994 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.319075 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.319174 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.319494 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.319549 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:10:12.320014 24157 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260812 00:10:12.320847 24650 raft_consensus.cc:348] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.321066 24650 raft_consensus.cc:374] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:12.321138 24650 raft_consensus.cc:729] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1a1ad80ebf4d6fa74b3c6020662589, State: Initialized, Role: FOLLOWER
I20260812 00:10:12.321077 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7225955d25c46e8a0d7b98372f77300. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:12.321192 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: This master's current role is: LEARNER
I20260812 00:10:12.321291 24650 consensus_queue.cc:260] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.321637 24650 ts_tablet_manager.cc:1428] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260812 00:10:12.321815 24620 heartbeater.cc:503] Master 127.16.99.254:40411 was elected leader, sending a full tablet report...
W20260812 00:10:12.364087 24480 tablet.cc:2369] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:10:12.460083 24338 tablet.cc:2369] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:12.516907 24657 raft_consensus.cc:482] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:12.517037 24657 raft_consensus.cc:504] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.517521 24657 leader_election.cc:288] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b376fa291da84a088858823be12e036d (127.16.99.194:44107), ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
I20260812 00:10:12.520707 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.517731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:40277 (local address 127.16.99.195:58045)
0812 00:10:12.517800 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.517813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.517869 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.518025 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.518026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.518028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:12.518111 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.518113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.518979 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.518980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.519837 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.519839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.520433 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.520434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.520512 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.520516 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.520586 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.520636 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 00:10:12.520732 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.517819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58045 (local address 127.16.99.193:40277)
0812 00:10:12.517871 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.517874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.517912 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:10:12.517942 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.517943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.517953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.518012 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:12.518191 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.518894 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.519930 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.520350 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.520406 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.520413 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.520607 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.520649 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:10:12.520900 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.517735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44107 (local address 127.16.99.195:40155)
0812 00:10:12.517883 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:12.517893 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:12.517933 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:12.518097 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:12.518097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:12.518099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:12.518178 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.518180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.519202 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.519203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.519950 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:12.519953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.520610 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.520611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:12.520693 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.520696 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:12.520756 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:10:12.520809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":90}
I20260812 00:10:12.520927 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.517889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40155 (local address 127.16.99.194:44107)
0812 00:10:12.517939 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.517941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.517958 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:10:12.518003 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.518004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.518015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.518073 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:12.518277 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.519116 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.520046 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.520530 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.520597 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.520605 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.520795 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.520851 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 00:10:12.521132 24287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f055da851846e380a8384b8dd66bcc" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa32349f97846248d6f3b2faa2ed898" is_pre_election: true
I20260812 00:10:12.521328 24287 raft_consensus.cc:2454] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 0.
I20260812 00:10:12.521389 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f055da851846e380a8384b8dd66bcc" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b376fa291da84a088858823be12e036d" is_pre_election: true
I20260812 00:10:12.521587 24430 raft_consensus.cc:2454] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 0.
I20260812 00:10:12.521605 24503 leader_election.cc:302] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae1a1ad80ebf4d6fa74b3c6020662589, ffa32349f97846248d6f3b2faa2ed898; no voters:
I20260812 00:10:12.521759 24657 raft_consensus.cc:2781] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:12.521833 24657 raft_consensus.cc:482] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:12.521870 24657 raft_consensus.cc:3037] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:12.522848 24657 raft_consensus.cc:504] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.523000 24657 leader_election.cc:288] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 election: Requested vote from peers b376fa291da84a088858823be12e036d (127.16.99.194:44107), ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
W20260812 00:10:12.523082 24622 tablet.cc:2369] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:12.523211 24287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f055da851846e380a8384b8dd66bcc" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa32349f97846248d6f3b2faa2ed898"
I20260812 00:10:12.523304 24287 raft_consensus.cc:3037] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:12.523490 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f055da851846e380a8384b8dd66bcc" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b376fa291da84a088858823be12e036d"
I20260812 00:10:12.523593 24430 raft_consensus.cc:3037] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:12.524165 24287 raft_consensus.cc:2454] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 1.
I20260812 00:10:12.524396 24430 raft_consensus.cc:2454] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 1.
I20260812 00:10:12.524601 24502 leader_election.cc:302] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae1a1ad80ebf4d6fa74b3c6020662589, b376fa291da84a088858823be12e036d; no voters:
I20260812 00:10:12.524698 24657 raft_consensus.cc:2781] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:12.524868 24657 raft_consensus.cc:686] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 LEADER]: Becoming Leader. State: Replica: ae1a1ad80ebf4d6fa74b3c6020662589, State: Running, Role: LEADER
I20260812 00:10:12.525084 24657 consensus_queue.cc:237] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:12.526177 23322 catalog_manager.cc:5495] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae1a1ad80ebf4d6fa74b3c6020662589 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } health_report { overall_health: UNKNOWN } } }
I20260812 00:10:12.526855 23911 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: NON_VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: true } } }
I20260812 00:10:12.526965 23911 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:12.527825 23911 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: attempting to promote NON_VOTER 9be67bdc0b864d89b09f4e1450f2ada3 to VOTER
I20260812 00:10:12.528002 23911 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } }
I20260812 00:10:12.528443 24157 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 LEARNER]: Refusing update from remote peer d7225955d25c46e8a0d7b98372f77300: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260812 00:10:12.528626 23963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Refusing update from remote peer d7225955d25c46e8a0d7b98372f77300: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260812 00:10:12.528776 24661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260812 00:10:12.528883 24663 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260812 00:10:12.529594 24663 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.529707 23963 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.529749 24157 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.530606 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.530659 23911 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.530696 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:12.530721 23911 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:12.530754 24663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.530817 24663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:12.530870 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.530967 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:10:12.530961 24664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7225955d25c46e8a0d7b98372f77300" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:12.531060 24664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:10:12.588145 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.585113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.16.99.195:44283)
0812 00:10:12.585275 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.585278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.585287 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:12.585342 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.585343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.585364 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.585429 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:12.585605 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.586112 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.587169 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.587408 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.587484 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.587716 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:12.587889 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:12.587968 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.588007 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.588028 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260812 00:10:12.590499 24430 raft_consensus.cc:1261] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Refusing update from remote peer ae1a1ad80ebf4d6fa74b3c6020662589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:12.590499 24287 raft_consensus.cc:1261] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 1 FOLLOWER]: Refusing update from remote peer ae1a1ad80ebf4d6fa74b3c6020662589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:12.590950 24657 consensus_queue.cc:1035] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:12.591302 24659 consensus_queue.cc:1035] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:12.595220 24666 mvcc.cc:205] Tried to move back new op lower bound from 7317477017966997504 to 7317477017703870464. Current Snapshot: MvccSnapshot[applied={T|T < 7317477017966997504}]
I20260812 00:10:12.595373 24668 mvcc.cc:205] Tried to move back new op lower bound from 7317477017966997504 to 7317477017703870464. Current Snapshot: MvccSnapshot[applied={T|T < 7317477017966997504}]
I20260812 00:10:12.599239 23322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:12.599372 23322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:12.601078 23322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7225955d25c46e8a0d7b98372f77300: Sending DeleteTablet for 3 replicas of tablet 32f055da851846e380a8384b8dd66bcc
I20260812 00:10:12.601481 24267 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f055da851846e380a8384b8dd66bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:12 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:49844
I20260812 00:10:12.601495 24410 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f055da851846e380a8384b8dd66bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:12 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:60692
I20260812 00:10:12.601889 24677 tablet_replica.cc:330] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: stopping tablet replica
I20260812 00:10:12.601903 24549 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f055da851846e380a8384b8dd66bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:12 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:33616
I20260812 00:10:12.602017 24677 raft_consensus.cc:2229] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:12.602138 24677 raft_consensus.cc:2258] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:12.602747 24677 ts_tablet_manager.cc:1905] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:12.603330 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 23288
I20260812 00:10:12.606941 24677 ts_tablet_manager.cc:1918] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:12.607151 24677 log.cc:1196] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/32f055da851846e380a8384b8dd66bcc
I20260812 00:10:12.607549 24677 ts_tablet_manager.cc:1939] T 32f055da851846e380a8384b8dd66bcc P b376fa291da84a088858823be12e036d: Deleting consensus metadata
I20260812 00:10:12.610505 24676 tablet_replica.cc:330] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: stopping tablet replica
I20260812 00:10:12.610690 24676 raft_consensus.cc:2229] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:12.610854 24676 raft_consensus.cc:2258] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:12.611093 24678 tablet_replica.cc:330] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: stopping tablet replica
I20260812 00:10:12.611220 24678 raft_consensus.cc:2229] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:12.611378 24678 raft_consensus.cc:2258] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:12.611631 24676 ts_tablet_manager.cc:1905] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:12.611894 24678 ts_tablet_manager.cc:1905] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 00:10:12.613335 24362 connection.cc:535] server connection from 127.0.0.1:60692 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:10:12.613793 24678 ts_tablet_manager.cc:1918] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:12.613880 24678 log.cc:1196] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/32f055da851846e380a8384b8dd66bcc
I20260812 00:10:12.614187 24678 ts_tablet_manager.cc:1939] T 32f055da851846e380a8384b8dd66bcc P ae1a1ad80ebf4d6fa74b3c6020662589: Deleting consensus metadata
I20260812 00:10:12.614812 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.602638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.16.99.252:42653)
0812 00:10:12.602682 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.602684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.602693 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:12.602717 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.602718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.602728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.602792 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.603734 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.604584 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.605799 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.606133 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.606182 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.608284 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.608286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.608286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.608296 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.610381 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.613515 (+ 3134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.613517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.613517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.613621 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.614030 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.614031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.614032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.614198 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.614294 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.614595 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.614709 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:10:12.615396 24503 connection.cc:441] server connection from 127.0.0.1:33616 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33616 (request call id 1) could send its response
I20260812 00:10:12.616652 24676 ts_tablet_manager.cc:1918] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:12.616840 24676 log.cc:1196] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/32f055da851846e380a8384b8dd66bcc
I20260812 00:10:12.617236 24676 ts_tablet_manager.cc:1939] T 32f055da851846e380a8384b8dd66bcc P ffa32349f97846248d6f3b2faa2ed898: Deleting consensus metadata
W20260812 00:10:12.617652 24216 connection.cc:441] server connection from 127.0.0.1:49844 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49844 (request call id 1) could send its response
I20260812 00:10:12.623386 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.615701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.16.99.252:42653)
0812 00:10:12.615743 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.615745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.615974 (+ 229us) server_negotiation.cc:407] Connection header received
0812 00:10:12.616086 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.616086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.616098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.616128 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.616899 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.617656 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.618521 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.618754 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.618790 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.620424 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.620425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.620426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.620435 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.622131 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.622559 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.622560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.622561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.622620 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.622847 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.622848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.622848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.622971 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.623080 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.623183 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.623267 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:10:12.624897 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:12.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.16.99.253:35763)
0812 00:10:12.615727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:12.615729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:12.616058 (+ 329us) server_negotiation.cc:407] Connection header received
0812 00:10:12.616202 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:12.616203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:12.616221 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:12.616255 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:12.617402 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.618290 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.619524 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:12.619887 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:12.620024 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:12.621394 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:12.621395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:12.621396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:12.621405 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:12.623288 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.624014 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.624015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.624015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.624073 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:12.624409 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:12.624410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:12.624410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:12.624522 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:12.624604 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:12.624737 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 00:10:12.624831 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:10:13.596288 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:13.595971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.193:45313)
0812 00:10:13.596130 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:13.596221 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
I20260812 00:10:13.596328 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:13.596035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.194:60823)
0812 00:10:13.596189 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:13.596276 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260812 00:10:13.596692 24478 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:40411 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
W20260812 00:10:13.596712 24336 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:40411 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
I20260812 00:10:13.597159 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:13.596773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.195:36405)
0812 00:10:13.597053 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:13.597116 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
W20260812 00:10:13.597334 24620 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:40411 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
I20260812 00:10:14.101835 24690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7225955d25c46e8a0d7b98372f77300)
I20260812 00:10:14.102021 24690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } }
I20260812 00:10:14.102201 24691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7225955d25c46e8a0d7b98372f77300)
I20260812 00:10:14.102401 24691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } }
I20260812 00:10:14.102512 24690 leader_election.cc:288] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411), 9b3c555aad11484eb965ff159e1b9f25 (127.16.99.253:35763)
I20260812 00:10:14.102725 24691 leader_election.cc:288] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411), 9be67bdc0b864d89b09f4e1450f2ada3 (127.16.99.252:42653)
I20260812 00:10:14.103116 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.102739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60468)
0812 00:10:14.102931 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.103061 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
I20260812 00:10:14.103235 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.102868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60476)
0812 00:10:14.103057 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.103129 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":60,"threads_started":1}
W20260812 00:10:14.103267 24134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
W20260812 00:10:14.103380 23939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
I20260812 00:10:14.103799 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.103710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60478)
0812 00:10:14.103751 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.103780 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 00:10:14.103920 24134 leader_election.cc:334] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411): Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
I20260812 00:10:14.104038 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.103818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60488)
0812 00:10:14.103932 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.103964 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260812 00:10:14.104288 23939 leader_election.cc:334] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411): Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
I20260812 00:10:14.106478 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.102907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.16.99.253:35763)
0812 00:10:14.103072 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:14.103075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:14.103085 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:14.103113 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:14.103115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:14.103126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:14.103406 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:14.103450 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.104333 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.105702 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.106085 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.106149 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.106183 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:14.106330 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 00:10:14.106398 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":55,"threads_started":1}
I20260812 00:10:14.106501 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.102702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:35763 (local address 127.0.0.1:58774)
0812 00:10:14.102876 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.102890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:14.102968 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:14.103247 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:14.103249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:14.103252 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:14.103330 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.103332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.104408 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.104409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.105597 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.105600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.106182 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.106184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.106259 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.106263 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:14.106309 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 00:10:14.106441 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":71,"thread_start_us":74,"threads_started":1}
I20260812 00:10:14.106757 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b3c555aad11484eb965ff159e1b9f25" is_pre_election: true
I20260812 00:10:14.106897 23963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be67bdc0b864d89b09f4e1450f2ada3 in term 1.
I20260812 00:10:14.107089 24131 leader_election.cc:302] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b3c555aad11484eb965ff159e1b9f25, 9be67bdc0b864d89b09f4e1450f2ada3; no voters: d7225955d25c46e8a0d7b98372f77300
I20260812 00:10:14.107239 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.103288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:42653 (local address 127.0.0.1:41080)
0812 00:10:14.103387 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.103397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:14.103456 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:14.104132 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:14.104134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:14.104136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:14.104177 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.104179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.104954 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.104955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.106091 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.106095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.106962 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.106963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.107048 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.107052 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:14.107109 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:10:14.107157 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
I20260812 00:10:14.107234 24690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 00:10:14.107250 24696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.103211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.16.99.252:42653)
0812 00:10:14.103953 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:14.103955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:14.103964 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:14.103986 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:14.103987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:14.103997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:14.104140 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:14.104315 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.104855 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.106216 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.106856 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.106944 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.106957 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:14.107116 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 00:10:14.107180 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":40,"threads_started":1}
I20260812 00:10:14.107275 24690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7225955d25c46e8a0d7b98372f77300)
I20260812 00:10:14.107298 24690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:10:14.107563 24157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b3c555aad11484eb965ff159e1b9f25" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" is_pre_election: true
I20260812 00:10:14.108304 24690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } }
I20260812 00:10:14.108530 24690 leader_election.cc:288] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 2 election: Requested vote from peers d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411), 9b3c555aad11484eb965ff159e1b9f25 (127.16.99.253:35763)
I20260812 00:10:14.108635 23963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9b3c555aad11484eb965ff159e1b9f25"
I20260812 00:10:14.108701 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.108545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60504)
0812 00:10:14.108634 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.108675 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 00:10:14.108732 23963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:10:14.109068 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.108957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60514)
0812 00:10:14.108993 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.109026 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 00:10:14.108505 24157 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b3c555aad11484eb965ff159e1b9f25 in current term 2: Already voted for candidate 9be67bdc0b864d89b09f4e1450f2ada3 in this term.
W20260812 00:10:14.109179 24134 leader_election.cc:334] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411): Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
I20260812 00:10:14.109342 23937 leader_election.cc:302] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b3c555aad11484eb965ff159e1b9f25; no voters: 9be67bdc0b864d89b09f4e1450f2ada3, d7225955d25c46e8a0d7b98372f77300
I20260812 00:10:14.109741 23963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be67bdc0b864d89b09f4e1450f2ada3 in term 2.
I20260812 00:10:14.109843 24691 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:10:14.109963 24131 leader_election.cc:302] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b3c555aad11484eb965ff159e1b9f25, 9be67bdc0b864d89b09f4e1450f2ada3; no voters: d7225955d25c46e8a0d7b98372f77300
I20260812 00:10:14.110077 24690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:10:14.110245 24690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [term 2 LEADER]: Becoming Leader. State: Replica: 9be67bdc0b864d89b09f4e1450f2ada3, State: Running, Role: LEADER
I20260812 00:10:14.110316 24690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } }
I20260812 00:10:14.111460 24701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9be67bdc0b864d89b09f4e1450f2ada3. Latest consensus state: current_term: 2 leader_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:14.111564 24701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:14.111768 24703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:14.112174 24703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:14.112226 24703 catalog_manager.cc:1259] Loaded cluster ID: 6322e762f280494b82b3a368b222fb97
I20260812 00:10:14.112242 24703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:14.112370 24703 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:14.112421 24703 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Loaded TSK: 0
I20260812 00:10:14.112509 24703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:10:14.115779 24145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:14.117555 24145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:14.118978 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.118833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60530)
0812 00:10:14.118902 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.118943 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 00:10:14.118996 23963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [term 2 FOLLOWER]: Refusing update from remote peer 9be67bdc0b864d89b09f4e1450f2ada3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 00:10:14.119216 24690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260812 00:10:14.119491 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.119397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.0.0.1:60536)
0812 00:10:14.119431 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.119467 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260812 00:10:14.119571 24134 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 -> Peer d7225955d25c46e8a0d7b98372f77300 (127.16.99.254:40411): Couldn't send request to peer d7225955d25c46e8a0d7b98372f77300. Status: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 00:10:14.119897 24691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9be67bdc0b864d89b09f4e1450f2ada3. Latest consensus state: current_term: 2 leader_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:14.120018 24691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:10:14.120404 24704 mvcc.cc:205] Tried to move back new op lower bound from 7317477024229052416 to 7317477024196677632. Current Snapshot: MvccSnapshot[applied={T|T < 7317477024229052416}]
I20260812 00:10:14.120483 24701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:14.120579 24701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:14.121205 24691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d7225955d25c46e8a0d7b98372f77300" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40411 } } peers { permanent_uuid: "9b3c555aad11484eb965ff159e1b9f25" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "9be67bdc0b864d89b09f4e1450f2ada3" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 42653 } attrs { promote: false } } }
I20260812 00:10:14.121352 24691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3c555aad11484eb965ff159e1b9f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:10:14.125950 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.122483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:44283 (local address 127.0.0.1:33632)
0812 00:10:14.122531 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.122544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:14.122586 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:14.122821 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:14.122822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:14.122824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:14.122882 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.122884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.123729 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.123730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.124547 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.124548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.125658 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.125738 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.125743 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:10:14.125797 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:10:14.125849 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 00:10:14.126040 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.122582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60704 (local address 127.16.99.194:44107)
0812 00:10:14.122712 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:14.122715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:14.123299 (+ 584us) server_negotiation.cc:407] Connection header received
0812 00:10:14.123343 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:14.123344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:14.123355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:14.123410 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:14.123562 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.124249 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125235 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.125670 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125741 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.125754 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:14.125902 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 00:10:14.125950 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260812 00:10:14.126087 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.122601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:40277 (local address 127.0.0.1:49860)
0812 00:10:14.122699 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.122708 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:14.122750 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:14.123106 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:14.123107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:14.123109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:14.123145 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.123146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.124068 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.124069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.125038 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.125041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.125805 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.125869 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.125871 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:10:14.125921 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 00:10:14.125957 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 00:10:14.126189 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.122558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.16.99.195:44283)
0812 00:10:14.122665 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:14.122667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:14.122676 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:14.122703 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:14.122704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:14.122714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:14.122816 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:14.122980 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.123623 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.124644 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.125570 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125647 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.125660 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:14.126018 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 00:10:14.126079 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260812 00:10:14.126436 24549 tablet_service.cc:1467] Processing CreateTablet for tablet 325f83a1ee1449e8b0183cad82a536f2 (DEFAULT_TABLE table=test-table [id=10b30b13b51d4fc98edb202e612a6ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:14.126479 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.122497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44107 (local address 127.0.0.1:60704)
0812 00:10:14.123155 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.123164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:14.123236 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:14.123453 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:14.123454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:14.123456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:14.123492 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.123493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.124347 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.124348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.125140 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:14.125141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.125745 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:14.125838 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.125842 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:14.125895 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 00:10:14.125942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":436,"thread_start_us":78,"threads_started":1}
I20260812 00:10:14.126592 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.122843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.16.99.193:40277)
0812 00:10:14.122968 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:14.122970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:14.122979 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:14.123006 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:14.123007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:14.123018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:14.123078 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:14.123221 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.123975 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125135 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.125699 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.125801 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.125815 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:14.126166 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 00:10:14.126516 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260812 00:10:14.126626 24549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325f83a1ee1449e8b0183cad82a536f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:14.126796 24267 tablet_service.cc:1467] Processing CreateTablet for tablet 325f83a1ee1449e8b0183cad82a536f2 (DEFAULT_TABLE table=test-table [id=10b30b13b51d4fc98edb202e612a6ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:14.126925 24267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325f83a1ee1449e8b0183cad82a536f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:14.128268 24715 tablet_bootstrap.cc:492] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: Bootstrap starting.
I20260812 00:10:14.128500 24410 tablet_service.cc:1467] Processing CreateTablet for tablet 325f83a1ee1449e8b0183cad82a536f2 (DEFAULT_TABLE table=test-table [id=10b30b13b51d4fc98edb202e612a6ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:14.128656 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325f83a1ee1449e8b0183cad82a536f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:14.128757 24715 tablet_bootstrap.cc:654] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:14.129110 24716 tablet_bootstrap.cc:492] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898: Bootstrap starting.
I20260812 00:10:14.129385 24715 tablet_bootstrap.cc:492] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: No bootstrap required, opened a new log
I20260812 00:10:14.129431 24715 ts_tablet_manager.cc:1397] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:14.129539 24715 raft_consensus.cc:348] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.129594 24715 raft_consensus.cc:374] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:14.129608 24715 raft_consensus.cc:729] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1a1ad80ebf4d6fa74b3c6020662589, State: Initialized, Role: FOLLOWER
I20260812 00:10:14.129669 24715 consensus_queue.cc:260] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.129884 24715 ts_tablet_manager.cc:1428] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:10:14.130095 24719 tablet_bootstrap.cc:492] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: Bootstrap starting.
I20260812 00:10:14.130326 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.130072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.195:38769)
0812 00:10:14.130231 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.130289 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260812 00:10:14.130686 24719 tablet_bootstrap.cc:654] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:14.130720 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.130554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.195:46635)
0812 00:10:14.130673 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.130698 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260812 00:10:14.131211 24716 tablet_bootstrap.cc:654] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:14.131587 24719 tablet_bootstrap.cc:492] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: No bootstrap required, opened a new log
I20260812 00:10:14.131649 24719 ts_tablet_manager.cc:1397] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:10:14.131872 24716 tablet_bootstrap.cc:492] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898: No bootstrap required, opened a new log
I20260812 00:10:14.131862 24719 raft_consensus.cc:348] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.131928 24716 ts_tablet_manager.cc:1397] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:10:14.131937 24719 raft_consensus.cc:374] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:14.131961 24719 raft_consensus.cc:729] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b376fa291da84a088858823be12e036d, State: Initialized, Role: FOLLOWER
I20260812 00:10:14.132019 24719 consensus_queue.cc:260] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.132084 24716 raft_consensus.cc:348] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.132159 24716 raft_consensus.cc:374] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:14.132181 24719 ts_tablet_manager.cc:1428] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:10:14.132191 24716 raft_consensus.cc:729] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa32349f97846248d6f3b2faa2ed898, State: Initialized, Role: FOLLOWER
I20260812 00:10:14.132265 24716 consensus_queue.cc:260] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.132652 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.132430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.194:48629)
0812 00:10:14.132564 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.132606 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260812 00:10:14.132682 24716 ts_tablet_manager.cc:1428] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:14.132668 24477 heartbeater.cc:503] Master 127.16.99.252:42653 was elected leader, sending a full tablet report...
I20260812 00:10:14.133342 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.133110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.193:44647)
0812 00:10:14.133250 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.133295 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":93,"threads_started":1}
I20260812 00:10:14.133448 24335 heartbeater.cc:503] Master 127.16.99.252:42653 was elected leader, sending a full tablet report...
I20260812 00:10:14.133687 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.133609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.193:46599)
0812 00:10:14.133639 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.133665 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260812 00:10:14.256345 24718 raft_consensus.cc:482] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:14.256492 24718 raft_consensus.cc:504] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.256752 24718 leader_election.cc:288] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b376fa291da84a088858823be12e036d (127.16.99.194:44107), ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
I20260812 00:10:14.257055 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325f83a1ee1449e8b0183cad82a536f2" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b376fa291da84a088858823be12e036d" is_pre_election: true
I20260812 00:10:14.257055 24287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325f83a1ee1449e8b0183cad82a536f2" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa32349f97846248d6f3b2faa2ed898" is_pre_election: true
I20260812 00:10:14.257181 24287 raft_consensus.cc:2454] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 0.
I20260812 00:10:14.257181 24430 raft_consensus.cc:2454] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 0.
I20260812 00:10:14.257383 24502 leader_election.cc:302] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae1a1ad80ebf4d6fa74b3c6020662589, b376fa291da84a088858823be12e036d; no voters:
I20260812 00:10:14.257524 24718 raft_consensus.cc:2781] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:14.257615 24718 raft_consensus.cc:482] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:14.257653 24718 raft_consensus.cc:3037] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:14.258602 24718 raft_consensus.cc:504] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.258774 24718 leader_election.cc:288] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 election: Requested vote from peers b376fa291da84a088858823be12e036d (127.16.99.194:44107), ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277)
I20260812 00:10:14.258968 24287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325f83a1ee1449e8b0183cad82a536f2" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa32349f97846248d6f3b2faa2ed898"
I20260812 00:10:14.259068 24287 raft_consensus.cc:3037] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:14.259053 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325f83a1ee1449e8b0183cad82a536f2" candidate_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b376fa291da84a088858823be12e036d"
I20260812 00:10:14.259196 24430 raft_consensus.cc:3037] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:14.259856 24287 raft_consensus.cc:2454] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 1.
I20260812 00:10:14.259865 24430 raft_consensus.cc:2454] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae1a1ad80ebf4d6fa74b3c6020662589 in term 1.
I20260812 00:10:14.260084 24502 leader_election.cc:302] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae1a1ad80ebf4d6fa74b3c6020662589, b376fa291da84a088858823be12e036d; no voters:
I20260812 00:10:14.260205 24718 raft_consensus.cc:2781] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:14.260377 24718 raft_consensus.cc:686] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 LEADER]: Becoming Leader. State: Replica: ae1a1ad80ebf4d6fa74b3c6020662589, State: Running, Role: LEADER
I20260812 00:10:14.260457 24619 heartbeater.cc:503] Master 127.16.99.252:42653 was elected leader, sending a full tablet report...
I20260812 00:10:14.260463 24718 consensus_queue.cc:237] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } }
I20260812 00:10:14.260784 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.260644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.195:46061)
0812 00:10:14.260699 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.260741 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260812 00:10:14.261418 24147 catalog_manager.cc:5495] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae1a1ad80ebf4d6fa74b3c6020662589 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1a1ad80ebf4d6fa74b3c6020662589" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 44283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 } health_report { overall_health: UNKNOWN } } }
I20260812 00:10:14.274770 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.271940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.16.99.195:44283)
0812 00:10:14.272011 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:14.272013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:14.272022 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:14.272050 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:14.272051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:14.272062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:14.272125 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:14.272402 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.272987 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.273859 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:14.274121 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:14.274219 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:14.274459 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:14.274506 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:14.274582 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:14.274646 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 00:10:14.274676 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:10:14.275939 24287 raft_consensus.cc:1261] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 1 FOLLOWER]: Refusing update from remote peer ae1a1ad80ebf4d6fa74b3c6020662589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:14.275980 24430 raft_consensus.cc:1261] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Refusing update from remote peer ae1a1ad80ebf4d6fa74b3c6020662589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:14.276264 24718 consensus_queue.cc:1035] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b376fa291da84a088858823be12e036d" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:14.276494 24729 consensus_queue.cc:1035] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa32349f97846248d6f3b2faa2ed898" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:14.277355 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.276924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.194:53005)
0812 00:10:14.277269 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.277314 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320}
I20260812 00:10:14.277988 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.277392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.193:58625)
0812 00:10:14.277905 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.277950 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438}
I20260812 00:10:14.278610 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.278454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.195:40807)
0812 00:10:14.278544 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.278575 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 00:10:14.279194 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.279050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.194:48151)
0812 00:10:14.279114 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.279150 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260812 00:10:14.280172 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.278879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40411 (local address 127.16.99.193:51049)
0812 00:10:14.280037 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.280070 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1139}
I20260812 00:10:14.280563 24733 mvcc.cc:205] Tried to move back new op lower bound from 7317477024871735296 to 7317477024811560960. Current Snapshot: MvccSnapshot[applied={T|T < 7317477024871735296}]
I20260812 00:10:14.280732 24147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:14.280825 24147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:14.282112 24147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9be67bdc0b864d89b09f4e1450f2ada3: Sending DeleteTablet for 3 replicas of tablet 325f83a1ee1449e8b0183cad82a536f2
I20260812 00:10:14.282478 24549 tablet_service.cc:1514] Processing DeleteTablet for tablet 325f83a1ee1449e8b0183cad82a536f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:14 UTC) from {username='oryx', principal='oryx/127.16.99.252@KRBTEST.COM'} at 127.0.0.1:33632
I20260812 00:10:14.282498 24267 tablet_service.cc:1514] Processing DeleteTablet for tablet 325f83a1ee1449e8b0183cad82a536f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:14 UTC) from {username='oryx', principal='oryx/127.16.99.252@KRBTEST.COM'} at 127.0.0.1:49860
I20260812 00:10:14.282691 24410 tablet_service.cc:1514] Processing DeleteTablet for tablet 325f83a1ee1449e8b0183cad82a536f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:14 UTC) from {username='oryx', principal='oryx/127.16.99.252@KRBTEST.COM'} at 127.0.0.1:60704
I20260812 00:10:14.282763 24742 tablet_replica.cc:330] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898: stopping tablet replica
I20260812 00:10:14.282820 24742 raft_consensus.cc:2229] T 325f83a1ee1449e8b0183cad82a536f2 P ffa32349f97846248d6f3b2faa2ed898 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:14.282944 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 24107
I20260812 00:10:14.284498 24743 tablet_replica.cc:330] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: stopping tablet replica
I20260812 00:10:14.284562 24743 raft_consensus.cc:2229] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:14.284571 24741 tablet_replica.cc:330] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: stopping tablet replica
I20260812 00:10:14.284621 24743 raft_consensus.cc:2258] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:14.284622 24741 raft_consensus.cc:2229] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:14.284698 24741 raft_consensus.cc:2258] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:14.284994 24743 ts_tablet_manager.cc:1905] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:14.285158 24741 ts_tablet_manager.cc:1905] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:14.286993 24741 ts_tablet_manager.cc:1918] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:14.287046 24741 log.cc:1196] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/325f83a1ee1449e8b0183cad82a536f2
I20260812 00:10:14.287263 24741 ts_tablet_manager.cc:1939] T 325f83a1ee1449e8b0183cad82a536f2 P ae1a1ad80ebf4d6fa74b3c6020662589: Deleting consensus metadata
I20260812 00:10:14.287472 24743 ts_tablet_manager.cc:1918] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:14.287552 24743 log.cc:1196] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/325f83a1ee1449e8b0183cad82a536f2
I20260812 00:10:14.287824 24743 ts_tablet_manager.cc:1939] T 325f83a1ee1449e8b0183cad82a536f2 P b376fa291da84a088858823be12e036d: Deleting consensus metadata
I20260812 00:10:14.288033 24134 catalog_manager.cc:4841] TS ae1a1ad80ebf4d6fa74b3c6020662589 (127.16.99.195:44283): tablet 325f83a1ee1449e8b0183cad82a536f2 (table test-table [id=10b30b13b51d4fc98edb202e612a6ad4]) successfully deleted
I20260812 00:10:14.288213 24133 catalog_manager.cc:4841] TS b376fa291da84a088858823be12e036d (127.16.99.194:44107): tablet 325f83a1ee1449e8b0183cad82a536f2 (table test-table [id=10b30b13b51d4fc98edb202e612a6ad4]) successfully deleted
I20260812 00:10:14.292834 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:14.292682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:40277 (local address 127.0.0.1:49868)
0812 00:10:14.292749 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:14.292787 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 00:10:14.292871 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 24343
W20260812 00:10:14.293038 24134 catalog_manager.cc:4584] TS ffa32349f97846248d6f3b2faa2ed898 (127.16.99.193:40277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 325f83a1ee1449e8b0183cad82a536f2: Network error: Client connection negotiation failed: client connection to 127.16.99.193:40277: connect: Connection refused (error 111)
I20260812 00:10:14.300278 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 24484
I20260812 00:10:14.308429 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 23809
I20260812 00:10:14.315586 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 24010
2026-08-12T00:10:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5062 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:14 dist-test-slave-zpfg krb5kdc[24746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:14 dist-test-slave-zpfg krb5kdc[24746](info): set up 2 sockets
Aug 12 00:10:14 dist-test-slave-zpfg krb5kdc[24746](info): commencing operation
krb5kdc: starting...
W20260812 00:10:15.431210 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:15 dist-test-slave-zpfg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab.
Entry for principal oryx/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab.
I20260812 00:10:15.570326 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42419
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab
--principal=oryx/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:42419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:15.656565 24759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:15.656798 24759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:15.656823 24759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:15.657994 24759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:15.658036 24759 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 00:10:15.658052 24759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:15.658066 24759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:15.658082 24759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:15.658095 24759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:15.659621 24759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:42419
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.254.keytab
--principal=oryx/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:15.659832 24759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:15.660012 24759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:15.665233 24765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:15.665270 24767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:15.665273 24764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:15.665484 24759 server_base.cc:1034] running on GCE node
I20260812 00:10:15.665763 24759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:15.666103 24759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:15.667268 24759 hybrid_clock.cc:648] HybridClock initialized: now 1786493415667256 us; error 34 us; skew 500 ppm
Aug 12 00:10:15 dist-test-slave-zpfg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493415, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:15.670127 24759 init.cc:377] Logged in from keytab as oryx/127.16.99.254@KRBTEST.COM (short username oryx)
I20260812 00:10:15.670701 24759 webserver.cc:460] Webserver started at http://127.16.99.254:41059/ using document root <none> and password file <none>
I20260812 00:10:15.670946 24759 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:15.671017 24759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:15.671177 24759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:15.672346 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "af90fc00a76b4d2985ee272ff5c093e6"
format_stamp: "Formatted at 2026-08-12 00:10:15 on dist-test-slave-zpfg"
server_key: "ce6a19617311b97c9c25d227dbbf49e9"
server_key_iv: "c9b301ec509a2b0e62a1106fb9885c61"
server_key_version: "encryptionkey@0"
I20260812 00:10:15.672746 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "af90fc00a76b4d2985ee272ff5c093e6"
format_stamp: "Formatted at 2026-08-12 00:10:15 on dist-test-slave-zpfg"
server_key: "ce6a19617311b97c9c25d227dbbf49e9"
server_key_iv: "c9b301ec509a2b0e62a1106fb9885c61"
server_key_version: "encryptionkey@0"
I20260812 00:10:15.674453 24759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:15.675552 24774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:15.675751 24759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:15.675896 24759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "af90fc00a76b4d2985ee272ff5c093e6"
format_stamp: "Formatted at 2026-08-12 00:10:15 on dist-test-slave-zpfg"
server_key: "ce6a19617311b97c9c25d227dbbf49e9"
server_key_iv: "c9b301ec509a2b0e62a1106fb9885c61"
server_key_version: "encryptionkey@0"
I20260812 00:10:15.675974 24759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:15.689400 24759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:15.691963 24759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:15.692119 24759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:15.697660 24759 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:42419
I20260812 00:10:15.697714 24840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:42419 every 8 connection(s)
I20260812 00:10:15.698130 24759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:15.698892 24841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:15.701241 24841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Bootstrap starting.
I20260812 00:10:15.701864 24841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:15.702114 24841 log.cc:824] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:15.702875 24841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: No bootstrap required, opened a new log
I20260812 00:10:15.704932 24841 raft_consensus.cc:348] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af90fc00a76b4d2985ee272ff5c093e6" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42419 } }
I20260812 00:10:15.705081 24841 raft_consensus.cc:374] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:15.705104 24841 raft_consensus.cc:729] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af90fc00a76b4d2985ee272ff5c093e6, State: Initialized, Role: FOLLOWER
I20260812 00:10:15.705261 24841 consensus_queue.cc:260] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af90fc00a76b4d2985ee272ff5c093e6" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42419 } }
I20260812 00:10:15.705350 24841 raft_consensus.cc:388] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:15.705385 24841 raft_consensus.cc:482] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:15.705439 24841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:15.705983 24841 raft_consensus.cc:504] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af90fc00a76b4d2985ee272ff5c093e6" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42419 } }
I20260812 00:10:15.706073 24841 leader_election.cc:302] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af90fc00a76b4d2985ee272ff5c093e6; no voters:
I20260812 00:10:15.706209 24841 leader_election.cc:288] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:15.706308 24845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:15.706542 24845 raft_consensus.cc:686] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [term 1 LEADER]: Becoming Leader. State: Replica: af90fc00a76b4d2985ee272ff5c093e6, State: Running, Role: LEADER
I20260812 00:10:15.706542 24841 sys_catalog.cc:564] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:15.706751 24845 consensus_queue.cc:237] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af90fc00a76b4d2985ee272ff5c093e6" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42419 } }
I20260812 00:10:15.706898 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 24759
I20260812 00:10:15.707007 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:15.707255 16783 external_mini_cluster.cc:1428] Setting key e440334b593b9356b60ff80df19563c3
I20260812 00:10:15.707475 24847 sys_catalog.cc:455] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af90fc00a76b4d2985ee272ff5c093e6. Latest consensus state: current_term: 1 leader_uuid: "af90fc00a76b4d2985ee272ff5c093e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af90fc00a76b4d2985ee272ff5c093e6" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42419 } } }
I20260812 00:10:15.707655 24847 sys_catalog.cc:458] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:15.708098 24856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:15.708453 24846 sys_catalog.cc:455] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af90fc00a76b4d2985ee272ff5c093e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af90fc00a76b4d2985ee272ff5c093e6" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42419 } } }
I20260812 00:10:15.708544 24846 sys_catalog.cc:458] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:15.708834 24856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:15.712824 24856 catalog_manager.cc:1347] Generated new cluster ID: 70befc487c6249c8911a460aae6d5f07
I20260812 00:10:15.712909 24856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:10:15 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:15.720501 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:15.708036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.16.99.254:42419)
0812 00:10:15.708299 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:15.708311 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:15.708353 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:10:15.708486 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:15.708495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:15.708564 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:15.708735 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:15.709400 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.711418 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.712232 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.712979 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.713116 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:15.716441 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:15.716446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:15.716446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:15.716460 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:15.718632 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:15.719107 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:15.719110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:15.719111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:15.719228 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:15.719470 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:15.719470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:15.719471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:15.719794 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:15.719944 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:15.720114 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 00:10:15.720257 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
I20260812 00:10:15.730979 24856 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:15.731843 24856 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:15.747422 24856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Generated new TSK 0
I20260812 00:10:15.747706 24856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab.
Entry for principal oryx/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab.
I20260812 00:10:15.783530 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42419
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab
--principal=oryx/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:15.882045 24869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:15.882326 24869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:15.882382 24869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:15.883826 24869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:15.883889 24869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:15.883916 24869 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:10:15.883939 24869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:15.885723 24869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab
--principal=oryx/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:42419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:15.885982 24869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:15.886207 24869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:15.892190 24877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:15.892510 24869 server_base.cc:1034] running on GCE node
W20260812 00:10:15.892190 24874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:15.892190 24875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:15.892920 24869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:15.893211 24869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:15.894425 24869 hybrid_clock.cc:648] HybridClock initialized: now 1786493415894408 us; error 29 us; skew 500 ppm
Aug 12 00:10:15 dist-test-slave-zpfg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493415, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:15.897413 24869 init.cc:377] Logged in from keytab as oryx/127.16.99.193@KRBTEST.COM (short username oryx)
I20260812 00:10:15.897853 24869 webserver.cc:460] Webserver started at http://127.16.99.193:34561/ using document root <none> and password file <none>
I20260812 00:10:15.898051 24869 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:15.898099 24869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:15.898202 24869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:15.899361 24869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "17508a2d617f4d4eb79e5f27d5447489"
format_stamp: "Formatted at 2026-08-12 00:10:15 on dist-test-slave-zpfg"
server_key: "f05286cf0967fb833470f042d2fffb9a"
server_key_iv: "8f726019700aec26fa581bff84ee121a"
server_key_version: "encryptionkey@0"
I20260812 00:10:15.899675 24869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "17508a2d617f4d4eb79e5f27d5447489"
format_stamp: "Formatted at 2026-08-12 00:10:15 on dist-test-slave-zpfg"
server_key: "f05286cf0967fb833470f042d2fffb9a"
server_key_iv: "8f726019700aec26fa581bff84ee121a"
server_key_version: "encryptionkey@0"
I20260812 00:10:15.901101 24869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:10:15.901932 24884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:15.902104 24869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 00:10:15.902187 24869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "17508a2d617f4d4eb79e5f27d5447489"
format_stamp: "Formatted at 2026-08-12 00:10:15 on dist-test-slave-zpfg"
server_key: "f05286cf0967fb833470f042d2fffb9a"
server_key_iv: "8f726019700aec26fa581bff84ee121a"
server_key_version: "encryptionkey@0"
I20260812 00:10:15.902302 24869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:15.919343 24869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:15.921998 24869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:15.922129 24869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:15.922830 24869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:15.922883 24869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:15.922907 24869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:15.922920 24869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:15.930932 24869 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:43335
I20260812 00:10:15.931031 25006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:43335 every 8 connection(s)
Aug 12 00:10:15 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493415, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:15.931371 24869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:15.935245 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:15.923368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:33853 (local address 127.16.99.254:42419)
0812 00:10:15.923450 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:15.923452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:15.923746 (+ 294us) server_negotiation.cc:407] Connection header received
0812 00:10:15.924562 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:15.924564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:15.924585 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:15.924666 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:15.925779 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.926646 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.928528 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.928845 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.928910 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:15.932018 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:15.932020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:15.932021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:15.932030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:15.933621 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:15.934112 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:15.934114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:15.934114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:15.934182 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:15.934535 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:15.934536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:15.934536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:15.934670 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:15.934750 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:15.935051 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 00:10:15.935148 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:10:15.935468 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:15.923341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42419 (local address 127.16.99.193:33853)
0812 00:10:15.923556 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:15.923614 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:15.924419 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:15.924849 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:15.924855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:15.925309 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:15.925656 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:15.925666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.926783 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.926785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:15.928411 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:15.928416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.928948 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.928949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:15.929141 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:15.929899 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:15.929910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:15.931905 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:15.933743 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:15.933754 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:15.933757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:15.934025 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:15.934259 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:15.934260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:15.934260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:15.934464 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:15.934742 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:15.934750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:15.934911 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 00:10:15.935027 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 00:10:15.935205 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260812 00:10:15.936101 25007 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42419
I20260812 00:10:15.936203 25007 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:15.936424 25007 heartbeater.cc:511] Master 127.16.99.254:42419 requested a full tablet report, sending...
I20260812 00:10:15.937024 24790 ts_manager.cc:194] Registered new tserver with Master: 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335)
I20260812 00:10:15.937914 24790 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:33853
I20260812 00:10:15.940740 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 24869
I20260812 00:10:15.940871 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:15.941099 16783 external_mini_cluster.cc:1428] Setting key da78ace5234dd1a91e5ada68f8d5d1b0
I20260812 00:10:15.944150 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:15.937434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:44583 (local address 127.16.99.254:42419)
0812 00:10:15.938131 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:15.938133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:15.938141 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:15.938166 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:15.938166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:15.938175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:15.938245 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:15.938666 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.939153 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.939864 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.940073 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.940153 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:15.941260 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:15.941262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:15.941262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:15.941270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:15.942850 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:15.943266 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:15.943268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:15.943268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:15.943330 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:15.943548 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:15.943549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:15.943549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:15.943675 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:15.943759 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:15.943940 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 00:10:15.944036 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677}
I20260812 00:10:15.944150 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:15.937338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42419 (local address 127.16.99.193:44583)
0812 00:10:15.937523 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:15.937534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:15.937587 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:15.938251 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:15.938252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:15.938522 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:15.938580 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:15.938582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.939248 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.939249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:15.939779 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:15.939780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:15.940150 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:15.940151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:15.940225 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:15.940714 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:15.940721 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:15.941147 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:15.942973 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:15.942975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:15.942975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:15.943185 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:15.943408 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:15.943409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:15.943410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:15.943478 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:15.943750 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:15.943751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:15.943807 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 00:10:15.943912 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:10:15.944036 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for oryx/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab.
Entry for principal oryx/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab.
I20260812 00:10:15.974500 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42419
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab
--principal=oryx/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:16.061198 25016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:16.061483 25016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:16.061532 25016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:16.062929 25016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:16.063001 25016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:16.063037 25016 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:10:16.063081 25016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:16.064762 25016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab
--principal=oryx/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:42419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:16.065068 25016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:16.065331 25016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:16.070717 25023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:16.070719 25025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:16.070719 25022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:16.071187 25016 server_base.cc:1034] running on GCE node
I20260812 00:10:16.071445 25016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:16.071703 25016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:16.072868 25016 hybrid_clock.cc:648] HybridClock initialized: now 1786493416072843 us; error 41 us; skew 500 ppm
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:16.075883 25016 init.cc:377] Logged in from keytab as oryx/127.16.99.194@KRBTEST.COM (short username oryx)
I20260812 00:10:16.076489 25016 webserver.cc:460] Webserver started at http://127.16.99.194:41963/ using document root <none> and password file <none>
I20260812 00:10:16.076767 25016 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:16.076845 25016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:16.077013 25016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:16.078184 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "685c39362ea4458da9e963663e307314"
format_stamp: "Formatted at 2026-08-12 00:10:16 on dist-test-slave-zpfg"
server_key: "aecbfde52f261ac08080e42434779a57"
server_key_iv: "d161f5e4a1101b84e62bf1be76dfccd1"
server_key_version: "encryptionkey@0"
I20260812 00:10:16.078668 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "685c39362ea4458da9e963663e307314"
format_stamp: "Formatted at 2026-08-12 00:10:16 on dist-test-slave-zpfg"
server_key: "aecbfde52f261ac08080e42434779a57"
server_key_iv: "d161f5e4a1101b84e62bf1be76dfccd1"
server_key_version: "encryptionkey@0"
I20260812 00:10:16.080294 25016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:10:16.081310 25032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.081658 25016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:16.081774 25016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "685c39362ea4458da9e963663e307314"
format_stamp: "Formatted at 2026-08-12 00:10:16 on dist-test-slave-zpfg"
server_key: "aecbfde52f261ac08080e42434779a57"
server_key_iv: "d161f5e4a1101b84e62bf1be76dfccd1"
server_key_version: "encryptionkey@0"
I20260812 00:10:16.081869 25016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:16.099360 25016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:16.101492 25016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:16.101682 25016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:16.102418 25016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:16.102474 25016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.102511 25016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:16.102525 25016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:16.110905 25016 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:37627
I20260812 00:10:16.111048 25151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:37627 every 8 connection(s)
I20260812 00:10:16.111392 25016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:10:16.113905 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.102879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:59473 (local address 127.16.99.254:42419)
0812 00:10:16.102956 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.102960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.103327 (+ 367us) server_negotiation.cc:407] Connection header received
0812 00:10:16.103904 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.103906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.103922 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.104008 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.104894 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.105580 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.107427 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.107682 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.107837 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.110656 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:16.110658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:16.110658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:16.110667 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:16.112386 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.112864 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.112866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.112867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.112931 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.113222 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.113223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.113223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.113350 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:16.113422 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.113716 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.113812 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:10:16.114107 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.102876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42419 (local address 127.16.99.194:59473)
0812 00:10:16.103069 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.103117 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.103757 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.104215 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.104220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.104532 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:16.104755 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.104760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.105691 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.105693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.107314 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.107322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.107872 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.107874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.108027 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.108634 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:16.108643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:16.110513 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:16.112522 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.112530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.112532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.112779 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.113002 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.113003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.113003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.113141 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.113425 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:16.113433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:16.113579 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.113693 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.113866 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260812 00:10:16.114656 25152 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42419
I20260812 00:10:16.114739 25152 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:16.114943 25152 heartbeater.cc:511] Master 127.16.99.254:42419 requested a full tablet report, sending...
I20260812 00:10:16.115392 24790 ts_manager.cc:194] Registered new tserver with Master: 685c39362ea4458da9e963663e307314 (127.16.99.194:37627)
I20260812 00:10:16.115796 24790 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:59473
I20260812 00:10:16.121598 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 25016
I20260812 00:10:16.121703 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:16.122000 16783 external_mini_cluster.cc:1428] Setting key 84e1d7cf050c30eaaaaace0e1e5db07d
I20260812 00:10:16.122262 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.115827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42419 (local address 127.16.99.194:43697)
0812 00:10:16.115987 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.116004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.116078 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.116646 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.116647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.116865 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:16.116925 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.116927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.117659 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.117660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.118210 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.118211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.118676 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.118677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.118732 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.119210 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:16.119214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:16.119512 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:16.120993 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.120994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.120995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.121169 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.121399 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.121400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.121400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.121474 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.121885 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:16.121886 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:16.121945 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.122029 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.122155 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260812 00:10:16.122296 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.115884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:43697 (local address 127.16.99.254:42419)
0812 00:10:16.115927 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.115929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.116053 (+ 124us) server_negotiation.cc:407] Connection header received
0812 00:10:16.116552 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.116553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.116564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.116633 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.117024 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.117522 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.118292 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.118555 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.118650 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.119597 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:16.119599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:16.119599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:16.119606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:16.120916 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.121234 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.121234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.121235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.121287 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.121542 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.121542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.121542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.121757 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:16.121872 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.122083 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.122187 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for oryx/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab.
Entry for principal oryx/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab.
I20260812 00:10:16.154675 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42419
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab
--principal=oryx/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:16.254751 25162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:16.255002 25162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:16.255031 25162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:16.256356 25162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:16.256407 25162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:16.256430 25162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 00:10:16.256450 25162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:16.258126 25162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab
--principal=oryx/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.99.254:42419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:16.258390 25162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:16.258615 25162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:16.263880 25169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:16.264062 25162 server_base.cc:1034] running on GCE node
W20260812 00:10:16.263919 25168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:16.263880 25171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:16.264546 25162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:16.264894 25162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:16.266057 25162 hybrid_clock.cc:648] HybridClock initialized: now 1786493416266048 us; error 57 us; skew 500 ppm
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:16.268985 25162 init.cc:377] Logged in from keytab as oryx/127.16.99.195@KRBTEST.COM (short username oryx)
I20260812 00:10:16.269500 25162 webserver.cc:460] Webserver started at http://127.16.99.195:34407/ using document root <none> and password file <none>
I20260812 00:10:16.269740 25162 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:16.269815 25162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:16.269976 25162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:16.271230 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "a2e00109249946c982787368b98b2e33"
format_stamp: "Formatted at 2026-08-12 00:10:16 on dist-test-slave-zpfg"
server_key: "e8e708ee4cc0ab7d31bb10438fc5fa18"
server_key_iv: "473dddcbdaeec7024c5426c3b341f697"
server_key_version: "encryptionkey@0"
I20260812 00:10:16.271638 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "a2e00109249946c982787368b98b2e33"
format_stamp: "Formatted at 2026-08-12 00:10:16 on dist-test-slave-zpfg"
server_key: "e8e708ee4cc0ab7d31bb10438fc5fa18"
server_key_iv: "473dddcbdaeec7024c5426c3b341f697"
server_key_version: "encryptionkey@0"
I20260812 00:10:16.273736 25162 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 00:10:16.274688 25178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.274902 25162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:16.275012 25162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "a2e00109249946c982787368b98b2e33"
format_stamp: "Formatted at 2026-08-12 00:10:16 on dist-test-slave-zpfg"
server_key: "e8e708ee4cc0ab7d31bb10438fc5fa18"
server_key_iv: "473dddcbdaeec7024c5426c3b341f697"
server_key_version: "encryptionkey@0"
I20260812 00:10:16.275101 25162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:16.293947 25162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:16.296561 25162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:16.296805 25162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:16.297425 25162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:16.297501 25162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.297555 25162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:16.297580 25162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:16.305902 25162 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:38133
I20260812 00:10:16.305989 25297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:38133 every 8 connection(s)
I20260812 00:10:16.306423 25162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:16.309376 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.298040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:44503 (local address 127.16.99.254:42419)
0812 00:10:16.298248 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.298251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.298525 (+ 274us) server_negotiation.cc:407] Connection header received
0812 00:10:16.299103 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.299105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.299121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.299206 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.300366 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.301265 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.302802 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.303014 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.303102 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.305864 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:16.305865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:16.305866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:16.305875 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:16.307812 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.308267 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.308269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.308269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.308342 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.308655 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.308656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.308657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.308797 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:16.308878 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.309181 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.309274 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:10:16.309593 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.298022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42419 (local address 127.16.99.195:44503)
0812 00:10:16.298331 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.298424 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.298986 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.299420 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.299425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.299860 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:16.300214 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.300223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.301402 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.301403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.302675 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.302679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.303137 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.303139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.303275 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.303910 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:16.303919 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:16.305745 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:16.307907 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.307915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.307918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.308179 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.308419 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.308420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.308420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.308576 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.308868 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:16.308876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:16.309036 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.309156 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.309349 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":100,"threads_started":1}
I20260812 00:10:16.310290 25298 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42419
I20260812 00:10:16.310410 25298 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:16.310622 25298 heartbeater.cc:511] Master 127.16.99.254:42419 requested a full tablet report, sending...
I20260812 00:10:16.311156 24792 ts_manager.cc:194] Registered new tserver with Master: a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.311528 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 25162
I20260812 00:10:16.311625 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:16.311725 24792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:44503
I20260812 00:10:16.311817 16783 external_mini_cluster.cc:1428] Setting key c2cd22c466ea81571b913a69a5efd032
I20260812 00:10:16.313455 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:10:16.319049 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.311537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42419 (local address 127.16.99.195:34441)
0812 00:10:16.311696 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.311715 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.311791 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.311991 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.311992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.312225 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:16.312308 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.312311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.313112 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.313113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.313867 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.313870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.314234 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.314235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.314324 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.314940 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:16.314947 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:16.315394 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:16.317516 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.317518 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.317519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.317791 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.318021 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.318022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.318023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.318209 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.318576 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:16.318577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:16.318639 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.318718 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.318865 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260812 00:10:16.319104 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.311591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:34441 (local address 127.16.99.254:42419)
0812 00:10:16.311636 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.311638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.311777 (+ 139us) server_negotiation.cc:407] Connection header received
0812 00:10:16.311899 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.311900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.311911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.311970 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.312401 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.312989 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.313962 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.314147 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.314210 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.315484 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:16.315485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:16.315485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:16.315491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:16.317404 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.317880 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.317881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.317882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.317939 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.318290 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.318292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.318292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.318502 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:16.318561 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.318755 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.318832 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:10:16.341084 24862 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:16.334652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.16.99.254:42419)
0812 00:10:16.334709 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.334712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.334723 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:16.334804 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.334805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.334816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.334901 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.335707 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.336346 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.337219 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.337457 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.337560 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.340985 (+ 3425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44038: BlockingRecv error: recv got EOF from 127.0.0.1:44038 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:16.350069 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.341973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.16.99.254:42419)
0812 00:10:16.342040 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.342041 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.342101 (+ 60us) server_negotiation.cc:407] Connection header received
0812 00:10:16.342201 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.342202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.342216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.342402 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.342966 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.343491 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.344187 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.344387 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.344479 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.347060 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:16.347062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:16.347062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:16.347071 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:16.348757 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.349220 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.349221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.349221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.349324 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.349582 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.349583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.349583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.349706 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:16.349786 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.349922 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.349998 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:10:16.352315 24792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:16.353979 24792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:16.361706 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.357059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.16.99.195:38133)
0812 00:10:16.357371 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.357375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.357383 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:16.357414 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.357416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.357427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.357520 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.357699 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.358873 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.360395 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.361094 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.361159 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.361226 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.361547 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.361593 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":85,"threads_started":1}
I20260812 00:10:16.361752 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.356939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:38133 (local address 127.0.0.1:46476)
0812 00:10:16.357168 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.357178 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.357220 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.357535 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.357538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.357540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.357613 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.357615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.358960 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.358961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.360289 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.360292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.361285 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.361287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.361392 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.361414 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.361519 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.361596 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":69,"thread_start_us":89,"threads_started":1}
I20260812 00:10:16.361749 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.356909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.16.99.194:37627)
0812 00:10:16.357702 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.357706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.357716 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:16.357756 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.357758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.357774 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.357849 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.358047 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.359117 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.360235 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.361041 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.361132 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.361226 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.361581 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.361628 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":99,"threads_started":1}
I20260812 00:10:16.362046 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.356691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37627 (local address 127.0.0.1:34430)
0812 00:10:16.357104 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.357152 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.357220 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.357921 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.357922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.357923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.357959 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.357960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.359217 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.359218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.360144 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.360146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.361287 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.361288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.361392 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.361414 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.361518 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.361991 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":426,"thread_start_us":94,"threads_started":1}
I20260812 00:10:16.362609 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.356732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43335 (local address 127.0.0.1:58970)
0812 00:10:16.357106 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.357152 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.357220 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.357674 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.357675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.357678 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.357740 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.357742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.358821 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.358822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.360309 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.360311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.362300 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.362302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.362414 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.362419 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.362470 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.362515 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":59,"thread_start_us":87,"threads_started":1}
I20260812 00:10:16.362890 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.356915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.16.99.193:43335)
0812 00:10:16.357405 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.357410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.357418 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:16.357449 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.357451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.357462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.357559 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.357844 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.358712 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.361271 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.362100 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.362171 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.362252 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.362729 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.362779 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":91,"threads_started":1}
I20260812 00:10:16.363080 25083 tablet_service.cc:1467] Processing CreateTablet for tablet 04ba46d222394cf88579e55f1ed7ad72 (DEFAULT_TABLE table=test-table [id=137e34db06274093a39d3956451aee06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.363544 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ba46d222394cf88579e55f1ed7ad72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.363477 24936 tablet_service.cc:1467] Processing CreateTablet for tablet 04ba46d222394cf88579e55f1ed7ad72 (DEFAULT_TABLE table=test-table [id=137e34db06274093a39d3956451aee06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.363757 24936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ba46d222394cf88579e55f1ed7ad72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.367309 25328 tablet_bootstrap.cc:492] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Bootstrap starting.
I20260812 00:10:16.367408 25330 tablet_bootstrap.cc:492] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Bootstrap starting.
I20260812 00:10:16.368042 25330 tablet_bootstrap.cc:654] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.368050 25229 tablet_service.cc:1467] Processing CreateTablet for tablet 04ba46d222394cf88579e55f1ed7ad72 (DEFAULT_TABLE table=test-table [id=137e34db06274093a39d3956451aee06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.368278 25328 tablet_bootstrap.cc:654] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.368449 25330 log.cc:824] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:16.368573 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ba46d222394cf88579e55f1ed7ad72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.368687 25328 log.cc:824] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:16.369354 25330 tablet_bootstrap.cc:492] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: No bootstrap required, opened a new log
I20260812 00:10:16.369455 25330 ts_tablet_manager.cc:1397] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:10:16.371071 25330 raft_consensus.cc:348] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.371245 25330 raft_consensus.cc:374] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.371300 25330 raft_consensus.cc:729] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c39362ea4458da9e963663e307314, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.371438 25330 consensus_queue.cc:260] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.371800 25330 ts_tablet_manager.cc:1428] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:10:16.371863 25152 heartbeater.cc:503] Master 127.16.99.254:42419 was elected leader, sending a full tablet report...
I20260812 00:10:16.372556 25328 tablet_bootstrap.cc:492] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: No bootstrap required, opened a new log
I20260812 00:10:16.372648 25334 tablet_bootstrap.cc:492] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Bootstrap starting.
I20260812 00:10:16.372702 25328 ts_tablet_manager.cc:1397] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 00:10:16.373378 25334 tablet_bootstrap.cc:654] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.373750 25334 log.cc:824] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:16.374655 25334 tablet_bootstrap.cc:492] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: No bootstrap required, opened a new log
I20260812 00:10:16.374737 25334 ts_tablet_manager.cc:1397] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:16.375244 25328 raft_consensus.cc:348] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.375448 25328 raft_consensus.cc:374] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.375486 25328 raft_consensus.cc:729] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.375617 25328 consensus_queue.cc:260] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.375919 25328 ts_tablet_manager.cc:1428] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:10:16.375988 25007 heartbeater.cc:503] Master 127.16.99.254:42419 was elected leader, sending a full tablet report...
I20260812 00:10:16.376844 25334 raft_consensus.cc:348] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.377005 25334 raft_consensus.cc:374] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.377029 25334 raft_consensus.cc:729] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e00109249946c982787368b98b2e33, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.377185 25334 consensus_queue.cc:260] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.377430 25334 ts_tablet_manager.cc:1428] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:10:16.377527 25298 heartbeater.cc:503] Master 127.16.99.254:42419 was elected leader, sending a full tablet report...
W20260812 00:10:16.432366 25008 tablet.cc:2369] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:16.435284 25336 raft_consensus.cc:482] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:16.435410 25336 raft_consensus.cc:504] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.435712 25336 leader_election.cc:288] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c39362ea4458da9e963663e307314 (127.16.99.194:37627), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.439531 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.435925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37627 (local address 127.16.99.193:39163)
0812 00:10:16.436130 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.436143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.436255 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.436448 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.436449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.436452 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.436531 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.436534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.437273 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.437274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.438653 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.438657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.439244 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.439245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.439311 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.439315 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.439391 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.439438 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":5,"thread_start_us":102,"threads_started":1}
I20260812 00:10:16.439519 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.436099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:39163 (local address 127.16.99.194:37627)
0812 00:10:16.436155 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.436157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.436249 (+ 92us) server_negotiation.cc:407] Connection header received
0812 00:10:16.436367 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.436369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.436382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.436442 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.436613 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.437172 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.438781 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.439160 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.439234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.439242 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.439402 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.439450 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 00:10:16.439996 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ba46d222394cf88579e55f1ed7ad72" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314" is_pre_election: true
I20260812 00:10:16.440187 25103 raft_consensus.cc:2454] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 0.
I20260812 00:10:16.440207 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.436369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:49255 (local address 127.16.99.195:38133)
0812 00:10:16.436415 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.436417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.436427 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:16.436479 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.436480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.436489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.436553 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.436721 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.437399 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.439280 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.439814 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.439916 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.439926 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.440094 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.440138 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:10:16.440240 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.436212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:38133 (local address 127.16.99.193:49255)
0812 00:10:16.436354 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.436365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.436422 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.436555 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.436556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.436557 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.436652 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.436653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.437485 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.437486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.438653 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.438657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.439916 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.439917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.440002 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.440005 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.440072 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.440117 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":52,"thread_start_us":69,"threads_started":1}
I20260812 00:10:16.440569 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ba46d222394cf88579e55f1ed7ad72" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33" is_pre_election: true
I20260812 00:10:16.440752 25250 raft_consensus.cc:2454] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 0.
I20260812 00:10:16.440970 24885 leader_election.cc:302] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, 685c39362ea4458da9e963663e307314; no voters:
I20260812 00:10:16.441100 25336 raft_consensus.cc:2781] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:16.441159 25336 raft_consensus.cc:482] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:16.441185 25336 raft_consensus.cc:3037] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.442099 25336 raft_consensus.cc:504] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.442248 25336 leader_election.cc:288] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 election: Requested vote from peers 685c39362ea4458da9e963663e307314 (127.16.99.194:37627), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.442469 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ba46d222394cf88579e55f1ed7ad72" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33"
I20260812 00:10:16.442469 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ba46d222394cf88579e55f1ed7ad72" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314"
I20260812 00:10:16.442559 25250 raft_consensus.cc:3037] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.442564 25103 raft_consensus.cc:3037] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.443499 25103 raft_consensus.cc:2454] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 1.
I20260812 00:10:16.443499 25250 raft_consensus.cc:2454] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 1.
I20260812 00:10:16.443681 24885 leader_election.cc:302] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, 685c39362ea4458da9e963663e307314; no voters:
I20260812 00:10:16.443791 25336 raft_consensus.cc:2781] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:16.443955 25336 raft_consensus.cc:686] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 LEADER]: Becoming Leader. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Running, Role: LEADER
I20260812 00:10:16.444036 25336 consensus_queue.cc:237] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } }
I20260812 00:10:16.444953 24792 catalog_manager.cc:5495] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193). New cstate: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } health_report { overall_health: HEALTHY } } }
I20260812 00:10:16.455787 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.453135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.16.99.193:43335)
0812 00:10:16.453205 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.453207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.453216 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:16.453256 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.453257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.453329 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.453415 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:16.453581 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.454064 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.454917 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.455135 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.455223 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.455427 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.455548 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.455647 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.455673 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.455696 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:10:16.457643 25250 raft_consensus.cc:1261] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Refusing update from remote peer 17508a2d617f4d4eb79e5f27d5447489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.457679 25103 raft_consensus.cc:1261] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Refusing update from remote peer 17508a2d617f4d4eb79e5f27d5447489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.458145 25336 consensus_queue.cc:1035] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.458292 25340 consensus_queue.cc:1035] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.461671 25343 mvcc.cc:205] Tried to move back new op lower bound from 7317477033807777792 to 7317477033755439104. Current Snapshot: MvccSnapshot[applied={T|T < 7317477033807777792}]
I20260812 00:10:16.461814 25344 mvcc.cc:205] Tried to move back new op lower bound from 7317477033807777792 to 7317477033755439104. Current Snapshot: MvccSnapshot[applied={T|T < 7317477033807777792}]
I20260812 00:10:16.462515 25346 mvcc.cc:205] Tried to move back new op lower bound from 7317477033807777792 to 7317477033755439104. Current Snapshot: MvccSnapshot[applied={T|T < 7317477033807777792}]
I20260812 00:10:16.464766 24790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.464879 24790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.465780 24790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Sending DeleteTablet for 3 replicas of tablet 04ba46d222394cf88579e55f1ed7ad72
I20260812 00:10:16.466172 25083 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ba46d222394cf88579e55f1ed7ad72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34430
I20260812 00:10:16.466200 25229 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ba46d222394cf88579e55f1ed7ad72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:46476
I20260812 00:10:16.466442 24790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 00:10:16.466509 24936 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ba46d222394cf88579e55f1ed7ad72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:58970
W20260812 00:10:16.466600 24790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:16.467338 25353 tablet_replica.cc:330] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: stopping tablet replica
I20260812 00:10:16.467655 25353 raft_consensus.cc:2229] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.468003 25229 tablet_service.cc:1467] Processing CreateTablet for tablet ad4a5659afd84d1ab916565f3a448aef (DEFAULT_TABLE table=test-table [id=dd56aeedb150434685799552871cd2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.468087 25353 raft_consensus.cc:2258] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.468084 24936 tablet_service.cc:1467] Processing CreateTablet for tablet ad4a5659afd84d1ab916565f3a448aef (DEFAULT_TABLE table=test-table [id=dd56aeedb150434685799552871cd2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.468153 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4a5659afd84d1ab916565f3a448aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.468194 24936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4a5659afd84d1ab916565f3a448aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.467115 25352 tablet_replica.cc:330] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: stopping tablet replica
I20260812 00:10:16.468292 25352 raft_consensus.cc:2229] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.468413 25352 raft_consensus.cc:2258] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.468912 25352 ts_tablet_manager.cc:1905] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.469540 25328 tablet_bootstrap.cc:492] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: Bootstrap starting.
I20260812 00:10:16.469565 25334 tablet_bootstrap.cc:492] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Bootstrap starting.
I20260812 00:10:16.469781 25353 ts_tablet_manager.cc:1905] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.470023 25334 tablet_bootstrap.cc:654] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.470019 25328 tablet_bootstrap.cc:654] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.470638 25083 tablet_service.cc:1467] Processing CreateTablet for tablet ad4a5659afd84d1ab916565f3a448aef (DEFAULT_TABLE table=test-table [id=dd56aeedb150434685799552871cd2d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.470774 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad4a5659afd84d1ab916565f3a448aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.471958 25330 tablet_bootstrap.cc:492] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Bootstrap starting.
I20260812 00:10:16.472425 25330 tablet_bootstrap.cc:654] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.472530 25334 tablet_bootstrap.cc:492] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: No bootstrap required, opened a new log
I20260812 00:10:16.472608 25334 ts_tablet_manager.cc:1397] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:10:16.472776 25334 raft_consensus.cc:348] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.472870 25334 raft_consensus.cc:374] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.472898 25334 raft_consensus.cc:729] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e00109249946c982787368b98b2e33, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.472944 25334 consensus_queue.cc:260] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.473033 25334 ts_tablet_manager.cc:1428] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:10:16.473068 25353 ts_tablet_manager.cc:1918] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:16.473160 25353 log.cc:1196] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/04ba46d222394cf88579e55f1ed7ad72
I20260812 00:10:16.473441 25353 ts_tablet_manager.cc:1939] T 04ba46d222394cf88579e55f1ed7ad72 P a2e00109249946c982787368b98b2e33: Deleting consensus metadata
I20260812 00:10:16.473883 24775 catalog_manager.cc:4841] TS a2e00109249946c982787368b98b2e33 (127.16.99.195:38133): tablet 04ba46d222394cf88579e55f1ed7ad72 (table test-table [id=137e34db06274093a39d3956451aee06]) successfully deleted
I20260812 00:10:16.473891 25352 ts_tablet_manager.cc:1918] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:16.473994 25352 log.cc:1196] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/04ba46d222394cf88579e55f1ed7ad72
I20260812 00:10:16.474277 25352 ts_tablet_manager.cc:1939] T 04ba46d222394cf88579e55f1ed7ad72 P 685c39362ea4458da9e963663e307314: Deleting consensus metadata
I20260812 00:10:16.474448 25354 tablet_replica.cc:330] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: stopping tablet replica
I20260812 00:10:16.474570 25354 raft_consensus.cc:2229] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:16.474736 25354 raft_consensus.cc:2258] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.475278 24775 catalog_manager.cc:4841] TS 685c39362ea4458da9e963663e307314 (127.16.99.194:37627): tablet 04ba46d222394cf88579e55f1ed7ad72 (table test-table [id=137e34db06274093a39d3956451aee06]) successfully deleted
I20260812 00:10:16.475595 25354 ts_tablet_manager.cc:1905] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.477357 25354 ts_tablet_manager.cc:1918] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:16.477440 25354 log.cc:1196] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/04ba46d222394cf88579e55f1ed7ad72
I20260812 00:10:16.477713 25354 ts_tablet_manager.cc:1939] T 04ba46d222394cf88579e55f1ed7ad72 P 17508a2d617f4d4eb79e5f27d5447489: Deleting consensus metadata
I20260812 00:10:16.478130 24778 catalog_manager.cc:4841] TS 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335): tablet 04ba46d222394cf88579e55f1ed7ad72 (table test-table [id=137e34db06274093a39d3956451aee06]) successfully deleted
I20260812 00:10:16.478874 25328 tablet_bootstrap.cc:492] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: No bootstrap required, opened a new log
I20260812 00:10:16.478945 25328 ts_tablet_manager.cc:1397] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260812 00:10:16.479058 25330 tablet_bootstrap.cc:492] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: No bootstrap required, opened a new log
I20260812 00:10:16.479121 25330 ts_tablet_manager.cc:1397] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260812 00:10:16.479143 25328 raft_consensus.cc:348] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.479241 25328 raft_consensus.cc:374] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.479276 25328 raft_consensus.cc:729] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.479327 25328 consensus_queue.cc:260] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.479295 25330 raft_consensus.cc:348] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.479398 25330 raft_consensus.cc:374] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.479418 25330 raft_consensus.cc:729] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c39362ea4458da9e963663e307314, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.479445 25328 ts_tablet_manager.cc:1428] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 00:10:16.479460 25330 consensus_queue.cc:260] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.479564 25330 ts_tablet_manager.cc:1428] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:10:16.497782 25336 raft_consensus.cc:482] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:16.497895 25336 raft_consensus.cc:504] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.498059 25336 leader_election.cc:288] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c39362ea4458da9e963663e307314 (127.16.99.194:37627), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.498293 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4a5659afd84d1ab916565f3a448aef" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314" is_pre_election: true
I20260812 00:10:16.498325 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4a5659afd84d1ab916565f3a448aef" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33" is_pre_election: true
I20260812 00:10:16.498456 25103 raft_consensus.cc:2454] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 0.
I20260812 00:10:16.498467 25250 raft_consensus.cc:2454] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 0.
I20260812 00:10:16.498613 24885 leader_election.cc:302] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, a2e00109249946c982787368b98b2e33; no voters:
I20260812 00:10:16.498755 25336 raft_consensus.cc:2781] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:16.498816 25336 raft_consensus.cc:482] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:16.498838 25336 raft_consensus.cc:3037] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.499780 25336 raft_consensus.cc:504] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.499935 25336 leader_election.cc:288] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 election: Requested vote from peers 685c39362ea4458da9e963663e307314 (127.16.99.194:37627), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.500150 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4a5659afd84d1ab916565f3a448aef" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314"
I20260812 00:10:16.500170 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad4a5659afd84d1ab916565f3a448aef" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33"
I20260812 00:10:16.500250 25250 raft_consensus.cc:3037] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.500252 25103 raft_consensus.cc:3037] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.501047 25103 raft_consensus.cc:2454] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 1.
I20260812 00:10:16.501055 25250 raft_consensus.cc:2454] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 1.
I20260812 00:10:16.501250 24885 leader_election.cc:302] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, a2e00109249946c982787368b98b2e33; no voters:
I20260812 00:10:16.501390 25336 raft_consensus.cc:2781] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:16.501449 25336 raft_consensus.cc:686] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 1 LEADER]: Becoming Leader. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Running, Role: LEADER
I20260812 00:10:16.501498 25336 consensus_queue.cc:237] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.502172 24792 catalog_manager.cc:5495] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193). New cstate: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } health_report { overall_health: UNKNOWN } } }
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493415, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.254@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:16.544691 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.535399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.16.99.254:42419)
0812 00:10:16.535464 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.535467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.535712 (+ 245us) server_negotiation.cc:407] Connection header received
0812 00:10:16.535772 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.535774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.535788 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.535877 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.536544 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.537332 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.538451 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.538717 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.538786 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.541566 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:16.541567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:16.541568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:16.541577 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:16.543348 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.543818 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.543819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.543819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.543880 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.544081 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.544082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.544082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.544225 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:16.544307 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.544478 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.544574 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:10:16.544802 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.535309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42419 (local address 127.0.0.1:44050)
0812 00:10:16.535548 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.535562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.535701 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.535916 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.535917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.536398 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:16.536453 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.536455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.537453 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.537454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.538318 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.538320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.538806 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.538808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.538875 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.539507 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:16.539513 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:16.541458 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:16.543470 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.543478 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.543481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.543717 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.543949 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:16.543950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:16.543951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:16.544015 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:16.544310 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:16.544319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:16.544367 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.544452 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.544659 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":83,"thread_start_us":121,"threads_started":1}
I20260812 00:10:16.547463 24792 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:44050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 00:10:16.547667 24792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:16.549412 24936 tablet_service.cc:1467] Processing CreateTablet for tablet dfe6a461f6404523aa08897f228630c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfaf9a40e5d24bca9f6276915a6cdd3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:10:16.549548 25229 tablet_service.cc:1467] Processing CreateTablet for tablet dfe6a461f6404523aa08897f228630c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfaf9a40e5d24bca9f6276915a6cdd3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:10:16.549608 24936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe6a461f6404523aa08897f228630c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.549553 25083 tablet_service.cc:1467] Processing CreateTablet for tablet dfe6a461f6404523aa08897f228630c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dfaf9a40e5d24bca9f6276915a6cdd3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 00:10:16.549678 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe6a461f6404523aa08897f228630c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.549747 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe6a461f6404523aa08897f228630c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.551332 25334 tablet_bootstrap.cc:492] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: Bootstrap starting.
I20260812 00:10:16.551553 25330 tablet_bootstrap.cc:492] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: Bootstrap starting.
I20260812 00:10:16.551792 25334 tablet_bootstrap.cc:654] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.552030 25330 tablet_bootstrap.cc:654] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.552138 25328 tablet_bootstrap.cc:492] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: Bootstrap starting.
I20260812 00:10:16.552850 25328 tablet_bootstrap.cc:654] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.553362 25334 tablet_bootstrap.cc:492] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: No bootstrap required, opened a new log
I20260812 00:10:16.553375 25330 tablet_bootstrap.cc:492] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: No bootstrap required, opened a new log
I20260812 00:10:16.553413 25334 ts_tablet_manager.cc:1397] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:10:16.553424 25330 ts_tablet_manager.cc:1397] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:10:16.553545 25334 raft_consensus.cc:348] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.553606 25334 raft_consensus.cc:374] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.553622 25334 raft_consensus.cc:729] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e00109249946c982787368b98b2e33, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.553581 25330 raft_consensus.cc:348] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.553668 25330 raft_consensus.cc:374] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.553690 25330 raft_consensus.cc:729] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c39362ea4458da9e963663e307314, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.553682 25334 consensus_queue.cc:260] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.553736 25330 consensus_queue.cc:260] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.553784 25334 ts_tablet_manager.cc:1428] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.553825 25330 ts_tablet_manager.cc:1428] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.553865 25337 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.553948 25337 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:10:16.554020 25333 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.554108 25333 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:10:16.554585 25328 tablet_bootstrap.cc:492] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: No bootstrap required, opened a new log
I20260812 00:10:16.554641 25328 ts_tablet_manager.cc:1397] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260812 00:10:16.554775 25328 raft_consensus.cc:348] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.554823 25328 raft_consensus.cc:374] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.554836 25328 raft_consensus.cc:729] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.554895 25328 consensus_queue.cc:260] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.555017 25328 ts_tablet_manager.cc:1428] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.555032 25336 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.555096 25336 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 00:10:16.557073 25299 tablet.cc:2369] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:16.584050 25336 raft_consensus.cc:482] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:16.584149 25336 raft_consensus.cc:504] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.584371 25336 leader_election.cc:288] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c39362ea4458da9e963663e307314 (127.16.99.194:37627), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.584630 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe6a461f6404523aa08897f228630c7" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314" is_pre_election: true
I20260812 00:10:16.584649 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe6a461f6404523aa08897f228630c7" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33" is_pre_election: true
I20260812 00:10:16.584739 25103 raft_consensus.cc:2454] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 0.
I20260812 00:10:16.584764 25250 raft_consensus.cc:2454] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 0.
I20260812 00:10:16.584946 24885 leader_election.cc:302] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, a2e00109249946c982787368b98b2e33; no voters:
I20260812 00:10:16.585067 25336 raft_consensus.cc:2781] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:16.585117 25336 raft_consensus.cc:482] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:16.585148 25336 raft_consensus.cc:3037] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.586130 25336 raft_consensus.cc:504] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.586263 25336 leader_election.cc:288] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 election: Requested vote from peers 685c39362ea4458da9e963663e307314 (127.16.99.194:37627), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.586517 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe6a461f6404523aa08897f228630c7" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314"
I20260812 00:10:16.586603 25103 raft_consensus.cc:3037] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.587371 25103 raft_consensus.cc:2454] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 1.
I20260812 00:10:16.587505 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe6a461f6404523aa08897f228630c7" candidate_uuid: "17508a2d617f4d4eb79e5f27d5447489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33"
I20260812 00:10:16.587565 24885 leader_election.cc:302] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, 685c39362ea4458da9e963663e307314; no voters:
I20260812 00:10:16.587596 25250 raft_consensus.cc:3037] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.588546 25250 raft_consensus.cc:2454] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17508a2d617f4d4eb79e5f27d5447489 in term 1.
I20260812 00:10:16.588668 25336 raft_consensus.cc:2781] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:16.588724 25336 raft_consensus.cc:686] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 LEADER]: Becoming Leader. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Running, Role: LEADER
I20260812 00:10:16.588768 25336 consensus_queue.cc:237] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.589032 25336 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: TxnStatusTablet state changed. Reason: New leader 17508a2d617f4d4eb79e5f27d5447489. Latest consensus state: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.589089 25336 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:10:16.589394 25369 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 00:10:16.589506 25369 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 00:10:16.589586 24792 catalog_manager.cc:5495] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193). New cstate: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } health_report { overall_health: UNKNOWN } } }
I20260812 00:10:16.603835 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.601372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.16.99.193:43335)
0812 00:10:16.601443 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.601446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.601456 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:16.601521 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.601523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.601536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.601581 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:16.601729 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.602208 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.603085 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.603303 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.603382 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.603556 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.603594 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.603665 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.603725 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.603751 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:10:16.603856 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.601274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43335 (local address 127.0.0.1:58978)
0812 00:10:16.601357 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.601377 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.601443 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.601601 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.601602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.601603 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:10:16.601659 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.601661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.602306 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.602307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.602969 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.602972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.603381 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.603382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.603452 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.603469 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.603666 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.603676 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.603724 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.603777 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:10:16.604835 25250 raft_consensus.cc:1261] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Refusing update from remote peer 17508a2d617f4d4eb79e5f27d5447489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.604838 25103 raft_consensus.cc:1261] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Refusing update from remote peer 17508a2d617f4d4eb79e5f27d5447489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.605099 25336 consensus_queue.cc:1035] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.605252 25340 consensus_queue.cc:1035] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.605801 25333 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: TxnStatusTablet state changed. Reason: New leader 17508a2d617f4d4eb79e5f27d5447489. Latest consensus state: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.605801 25337 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: TxnStatusTablet state changed. Reason: New leader 17508a2d617f4d4eb79e5f27d5447489. Latest consensus state: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.605921 25337 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:10:16.605921 25333 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:10:16.606858 25336 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.606957 25336 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:10:16.607309 25337 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.607403 25337 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P a2e00109249946c982787368b98b2e33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 00:10:16.607750 25340 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.607839 25340 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P 17508a2d617f4d4eb79e5f27d5447489: This TxnStatusTablet replica's current role is: LEADER
I20260812 00:10:16.608377 25333 tablet_replica.cc:439] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17508a2d617f4d4eb79e5f27d5447489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } } }
I20260812 00:10:16.608464 25333 tablet_replica.cc:442] T dfe6a461f6404523aa08897f228630c7 P 685c39362ea4458da9e963663e307314: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 00:10:16.612632 25153 tablet.cc:2369] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:16.613920 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.611120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:56957 (local address 127.16.99.193:43335)
0812 00:10:16.611181 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.611184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.611343 (+ 159us) server_negotiation.cc:407] Connection header received
0812 00:10:16.611401 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.611402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.611410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.611451 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:16.611611 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.612093 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.613228 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.613463 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.613518 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.613710 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.613736 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.613774 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.613853 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.613869 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:10:16.614017 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.611002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43335 (local address 127.16.99.193:56957)
0812 00:10:16.611266 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.611277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.611343 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.611464 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.611465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.611466 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:10:16.611518 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.611519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.612174 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.612175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.613131 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.613134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.613539 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.613540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.613604 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.613628 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.613796 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.613798 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.613852 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.613904 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":138,"threads_started":1}
I20260812 00:10:16.616555 25250 raft_consensus.cc:1261] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Refusing update from remote peer 17508a2d617f4d4eb79e5f27d5447489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.616555 25103 raft_consensus.cc:1261] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Refusing update from remote peer 17508a2d617f4d4eb79e5f27d5447489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.616775 25340 consensus_queue.cc:1035] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.616896 25336 consensus_queue.cc:1035] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.631160 24792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.631292 24792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.631994 24792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Sending DeleteTablet for 3 replicas of tablet ad4a5659afd84d1ab916565f3a448aef
I20260812 00:10:16.632242 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4a5659afd84d1ab916565f3a448aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:58970
I20260812 00:10:16.632254 25083 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4a5659afd84d1ab916565f3a448aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34430
I20260812 00:10:16.632360 25352 tablet_replica.cc:330] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: stopping tablet replica
I20260812 00:10:16.632368 25354 tablet_replica.cc:330] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: stopping tablet replica
I20260812 00:10:16.632405 25352 raft_consensus.cc:2229] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.632427 25354 raft_consensus.cc:2229] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:16.632443 25352 raft_consensus.cc:2258] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.632423 25229 tablet_service.cc:1514] Processing DeleteTablet for tablet ad4a5659afd84d1ab916565f3a448aef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:46476
I20260812 00:10:16.632485 25354 raft_consensus.cc:2258] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.632503 25353 tablet_replica.cc:330] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: stopping tablet replica
I20260812 00:10:16.632551 25353 raft_consensus.cc:2229] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.632591 25353 raft_consensus.cc:2258] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.632953 25352 ts_tablet_manager.cc:1905] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.632966 25354 ts_tablet_manager.cc:1905] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.634950 25353 ts_tablet_manager.cc:1905] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.635012 25352 ts_tablet_manager.cc:1918] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:10:16.635076 25352 log.cc:1196] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/ad4a5659afd84d1ab916565f3a448aef
I20260812 00:10:16.635331 25352 ts_tablet_manager.cc:1939] T ad4a5659afd84d1ab916565f3a448aef P 685c39362ea4458da9e963663e307314: Deleting consensus metadata
I20260812 00:10:16.636579 25353 ts_tablet_manager.cc:1918] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:10:16.636665 25353 log.cc:1196] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/ad4a5659afd84d1ab916565f3a448aef
I20260812 00:10:16.636855 25354 ts_tablet_manager.cc:1918] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:10:16.636889 25353 ts_tablet_manager.cc:1939] T ad4a5659afd84d1ab916565f3a448aef P a2e00109249946c982787368b98b2e33: Deleting consensus metadata
I20260812 00:10:16.636919 25354 log.cc:1196] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/ad4a5659afd84d1ab916565f3a448aef
I20260812 00:10:16.637153 25354 ts_tablet_manager.cc:1939] T ad4a5659afd84d1ab916565f3a448aef P 17508a2d617f4d4eb79e5f27d5447489: Deleting consensus metadata
I20260812 00:10:16.637519 24778 catalog_manager.cc:4841] TS 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335): tablet ad4a5659afd84d1ab916565f3a448aef (table test-table [id=dd56aeedb150434685799552871cd2d6]) successfully deleted
I20260812 00:10:16.637867 24775 catalog_manager.cc:4841] TS a2e00109249946c982787368b98b2e33 (127.16.99.195:38133): tablet ad4a5659afd84d1ab916565f3a448aef (table test-table [id=dd56aeedb150434685799552871cd2d6]) successfully deleted
I20260812 00:10:16.638049 24775 catalog_manager.cc:4841] TS 685c39362ea4458da9e963663e307314 (127.16.99.194:37627): tablet ad4a5659afd84d1ab916565f3a448aef (table test-table [id=dd56aeedb150434685799552871cd2d6]) successfully deleted
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:10:16.659832 24862 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:16.653275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.16.99.254:42419)
0812 00:10:16.653338 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.653341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.653451 (+ 110us) server_negotiation.cc:407] Connection header received
0812 00:10:16.653528 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.653529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.653538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.653592 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.654382 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.654890 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.656022 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.656262 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.656336 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.659764 (+ 3428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44066: BlockingRecv error: recv got EOF from 127.0.0.1:44066 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 12 00:10:16 dist-test-slave-zpfg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM
I20260812 00:10:16.668934 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.660921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.16.99.254:42419)
0812 00:10:16.660984 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.660987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.661048 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:10:16.661111 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.661112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.661123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.661180 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:16.661693 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.662474 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.663174 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.663517 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.663599 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.665975 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:16.665977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:16.665977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:16.665986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:16.667638 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.667962 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.667963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.667964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.668055 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:16.668374 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:16.668375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:16.668375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:16.668528 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:16.668642 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.668750 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.668834 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 00:10:16.670784 24792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:16.670982 24792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:16.672451 25229 tablet_service.cc:1467] Processing CreateTablet for tablet ca6037053b3a487da67e710e12bb09e0 (DEFAULT_TABLE table=test-table [id=e323429fbb4343a3aad0e17856e7f7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.672441 24935 tablet_service.cc:1467] Processing CreateTablet for tablet ca6037053b3a487da67e710e12bb09e0 (DEFAULT_TABLE table=test-table [id=e323429fbb4343a3aad0e17856e7f7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.672499 25083 tablet_service.cc:1467] Processing CreateTablet for tablet ca6037053b3a487da67e710e12bb09e0 (DEFAULT_TABLE table=test-table [id=e323429fbb4343a3aad0e17856e7f7a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.672590 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6037053b3a487da67e710e12bb09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.672592 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6037053b3a487da67e710e12bb09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.672652 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6037053b3a487da67e710e12bb09e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.674006 25330 tablet_bootstrap.cc:492] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: Bootstrap starting.
I20260812 00:10:16.674122 25334 tablet_bootstrap.cc:492] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: Bootstrap starting.
I20260812 00:10:16.674547 25330 tablet_bootstrap.cc:654] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.674882 25334 tablet_bootstrap.cc:654] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.675268 25328 tablet_bootstrap.cc:492] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: Bootstrap starting.
I20260812 00:10:16.675357 25330 tablet_bootstrap.cc:492] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: No bootstrap required, opened a new log
I20260812 00:10:16.675420 25330 ts_tablet_manager.cc:1397] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:16.675608 25328 tablet_bootstrap.cc:654] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.675580 25330 raft_consensus.cc:348] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.675664 25330 raft_consensus.cc:374] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.675685 25330 raft_consensus.cc:729] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c39362ea4458da9e963663e307314, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.675740 25330 consensus_queue.cc:260] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.675832 25330 ts_tablet_manager.cc:1428] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 00:10:16.676210 25334 tablet_bootstrap.cc:492] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: No bootstrap required, opened a new log
I20260812 00:10:16.676283 25334 ts_tablet_manager.cc:1397] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:10:16.676460 25334 raft_consensus.cc:348] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.676541 25334 raft_consensus.cc:374] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.676566 25334 raft_consensus.cc:729] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e00109249946c982787368b98b2e33, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.676614 25334 consensus_queue.cc:260] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.676787 25334 ts_tablet_manager.cc:1428] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.677278 25328 tablet_bootstrap.cc:492] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: No bootstrap required, opened a new log
I20260812 00:10:16.677331 25328 ts_tablet_manager.cc:1397] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 00:10:16.677455 25328 raft_consensus.cc:348] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.677557 25328 raft_consensus.cc:374] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.677582 25328 raft_consensus.cc:729] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.677626 25328 consensus_queue.cc:260] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.677721 25328 ts_tablet_manager.cc:1428] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.784266 25337 raft_consensus.cc:482] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:16.784406 25337 raft_consensus.cc:504] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.784817 25337 leader_election.cc:288] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335), 685c39362ea4458da9e963663e307314 (127.16.99.194:37627)
I20260812 00:10:16.787969 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.785066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:38283 (local address 127.16.99.194:37627)
0812 00:10:16.785120 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.785122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.785133 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:16.785229 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.785230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.785239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.785297 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.785492 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.786008 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.787183 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.787571 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.787665 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.787675 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.787858 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.787902 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 00:10:16.787972 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.785008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37627 (local address 127.16.99.195:38283)
0812 00:10:16.785082 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.785093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.785141 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.785331 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.785332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.785334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.785407 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.785409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.786099 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.786100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.787076 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.787079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.787663 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.787664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.787760 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.787763 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.787848 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.787891 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":32}
I20260812 00:10:16.788064 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.785133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58499 (local address 127.16.99.193:43335)
0812 00:10:16.785196 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.785198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.785338 (+ 140us) server_negotiation.cc:407] Connection header received
0812 00:10:16.785430 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.785431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.785441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.785499 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.785694 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.786215 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.787270 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.787648 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.787737 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.787749 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.787941 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.787981 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 00:10:16.788277 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6037053b3a487da67e710e12bb09e0" candidate_uuid: "a2e00109249946c982787368b98b2e33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314" is_pre_election: true
I20260812 00:10:16.788336 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.785010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43335 (local address 127.16.99.195:58499)
0812 00:10:16.785243 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.785254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.785331 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.785503 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.785503 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.785505 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.785596 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.785598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.786371 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.786372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.787170 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.787172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.787752 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.787753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.787848 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.787852 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.787930 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.787976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":112,"thread_start_us":74,"threads_started":1}
I20260812 00:10:16.788393 25103 raft_consensus.cc:2454] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e00109249946c982787368b98b2e33 in term 0.
I20260812 00:10:16.788630 25179 leader_election.cc:302] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c39362ea4458da9e963663e307314, a2e00109249946c982787368b98b2e33; no voters:
I20260812 00:10:16.788729 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6037053b3a487da67e710e12bb09e0" candidate_uuid: "a2e00109249946c982787368b98b2e33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17508a2d617f4d4eb79e5f27d5447489" is_pre_election: true
I20260812 00:10:16.788836 25337 raft_consensus.cc:2781] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:16.788882 25337 raft_consensus.cc:482] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:16.788901 25337 raft_consensus.cc:3037] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.788897 24956 raft_consensus.cc:2454] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e00109249946c982787368b98b2e33 in term 0.
I20260812 00:10:16.789707 25337 raft_consensus.cc:504] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.789872 25337 leader_election.cc:288] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [CANDIDATE]: Term 1 election: Requested vote from peers 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335), 685c39362ea4458da9e963663e307314 (127.16.99.194:37627)
I20260812 00:10:16.790045 25103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6037053b3a487da67e710e12bb09e0" candidate_uuid: "a2e00109249946c982787368b98b2e33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c39362ea4458da9e963663e307314"
I20260812 00:10:16.790150 25103 raft_consensus.cc:3037] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.790442 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6037053b3a487da67e710e12bb09e0" candidate_uuid: "a2e00109249946c982787368b98b2e33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17508a2d617f4d4eb79e5f27d5447489"
I20260812 00:10:16.790521 24956 raft_consensus.cc:3037] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.790915 25103 raft_consensus.cc:2454] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e00109249946c982787368b98b2e33 in term 1.
I20260812 00:10:16.791119 25179 leader_election.cc:302] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c39362ea4458da9e963663e307314, a2e00109249946c982787368b98b2e33; no voters:
I20260812 00:10:16.791234 25337 raft_consensus.cc:2781] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:16.791378 24956 raft_consensus.cc:2454] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e00109249946c982787368b98b2e33 in term 1.
I20260812 00:10:16.791450 25337 raft_consensus.cc:686] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 1 LEADER]: Becoming Leader. State: Replica: a2e00109249946c982787368b98b2e33, State: Running, Role: LEADER
I20260812 00:10:16.791528 25337 consensus_queue.cc:237] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.792320 24792 catalog_manager.cc:5495] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e00109249946c982787368b98b2e33 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "a2e00109249946c982787368b98b2e33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } health_report { overall_health: HEALTHY } } }
I20260812 00:10:16.826292 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.823570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.16.99.195:38133)
0812 00:10:16.823637 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.823639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.823647 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:16.823697 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.823699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.823723 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.823822 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:16.823963 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.824593 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.825330 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.825590 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.825671 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.825804 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.825935 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.825992 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.826031 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.826051 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:10:16.827430 25103 raft_consensus.cc:1261] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Refusing update from remote peer a2e00109249946c982787368b98b2e33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.827497 24956 raft_consensus.cc:1261] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Refusing update from remote peer a2e00109249946c982787368b98b2e33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.827883 25337 consensus_queue.cc:1035] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.828050 25398 consensus_queue.cc:1035] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.831789 24792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.831900 24792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.832610 24792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Sending DeleteTablet for 3 replicas of tablet ca6037053b3a487da67e710e12bb09e0
I20260812 00:10:16.832888 25083 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6037053b3a487da67e710e12bb09e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34430
I20260812 00:10:16.832883 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6037053b3a487da67e710e12bb09e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:58970
I20260812 00:10:16.832985 25354 tablet_replica.cc:330] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: stopping tablet replica
I20260812 00:10:16.832985 25352 tablet_replica.cc:330] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: stopping tablet replica
I20260812 00:10:16.833048 25354 raft_consensus.cc:2229] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.833041 25352 raft_consensus.cc:2229] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.833096 25352 raft_consensus.cc:2258] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.833098 25354 raft_consensus.cc:2258] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.833370 25229 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6037053b3a487da67e710e12bb09e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:46476
I20260812 00:10:16.833467 25353 tablet_replica.cc:330] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: stopping tablet replica
I20260812 00:10:16.833523 25353 raft_consensus.cc:2229] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:16.833575 25354 ts_tablet_manager.cc:1905] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.833618 25353 raft_consensus.cc:2258] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.833892 25353 ts_tablet_manager.cc:1905] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.833899 24792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:16.834033 24792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:16.835227 24935 tablet_service.cc:1467] Processing CreateTablet for tablet fd8b701bf40748a69871ad07b06d8e87 (DEFAULT_TABLE table=test-table [id=2502026236b84fc3bdf4464d4078e9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.835650 25083 tablet_service.cc:1467] Processing CreateTablet for tablet fd8b701bf40748a69871ad07b06d8e87 (DEFAULT_TABLE table=test-table [id=2502026236b84fc3bdf4464d4078e9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.835683 25354 ts_tablet_manager.cc:1918] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:16.835733 25354 log.cc:1196] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/ca6037053b3a487da67e710e12bb09e0
I20260812 00:10:16.835774 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8b701bf40748a69871ad07b06d8e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.835834 25353 ts_tablet_manager.cc:1918] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:16.835889 25353 log.cc:1196] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/ca6037053b3a487da67e710e12bb09e0
I20260812 00:10:16.835970 25354 ts_tablet_manager.cc:1939] T ca6037053b3a487da67e710e12bb09e0 P 17508a2d617f4d4eb79e5f27d5447489: Deleting consensus metadata
I20260812 00:10:16.836205 25353 ts_tablet_manager.cc:1939] T ca6037053b3a487da67e710e12bb09e0 P a2e00109249946c982787368b98b2e33: Deleting consensus metadata
I20260812 00:10:16.837026 25330 tablet_bootstrap.cc:492] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: Bootstrap starting.
I20260812 00:10:16.837164 25229 tablet_service.cc:1467] Processing CreateTablet for tablet fd8b701bf40748a69871ad07b06d8e87 (DEFAULT_TABLE table=test-table [id=2502026236b84fc3bdf4464d4078e9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:16.837276 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8b701bf40748a69871ad07b06d8e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.837716 25330 tablet_bootstrap.cc:654] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.838376 25334 tablet_bootstrap.cc:492] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: Bootstrap starting.
I20260812 00:10:16.838423 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8b701bf40748a69871ad07b06d8e87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:16.838457 25330 tablet_bootstrap.cc:492] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: No bootstrap required, opened a new log
I20260812 00:10:16.838502 25330 ts_tablet_manager.cc:1397] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:16.838642 25330 raft_consensus.cc:348] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.838711 25330 raft_consensus.cc:374] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.838732 25330 raft_consensus.cc:729] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c39362ea4458da9e963663e307314, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.838776 25330 consensus_queue.cc:260] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.838807 25334 tablet_bootstrap.cc:654] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.838864 25330 ts_tablet_manager.cc:1428] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.839435 25334 tablet_bootstrap.cc:492] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: No bootstrap required, opened a new log
I20260812 00:10:16.839483 25334 ts_tablet_manager.cc:1397] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:16.839632 25334 raft_consensus.cc:348] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.839700 25334 raft_consensus.cc:374] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.839720 25334 raft_consensus.cc:729] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e00109249946c982787368b98b2e33, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.839749 25334 consensus_queue.cc:260] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.839849 25334 ts_tablet_manager.cc:1428] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.839915 25328 tablet_bootstrap.cc:492] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489: Bootstrap starting.
I20260812 00:10:16.840215 25328 tablet_bootstrap.cc:654] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:16.840273 24778 catalog_manager.cc:4841] TS 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335): tablet ca6037053b3a487da67e710e12bb09e0 (table test-table [id=e323429fbb4343a3aad0e17856e7f7a4]) successfully deleted
I20260812 00:10:16.840570 24775 catalog_manager.cc:4841] TS a2e00109249946c982787368b98b2e33 (127.16.99.195:38133): tablet ca6037053b3a487da67e710e12bb09e0 (table test-table [id=e323429fbb4343a3aad0e17856e7f7a4]) successfully deleted
I20260812 00:10:16.840966 25352 ts_tablet_manager.cc:1905] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.842435 25352 ts_tablet_manager.cc:1918] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:16.842499 25352 log.cc:1196] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/ca6037053b3a487da67e710e12bb09e0
I20260812 00:10:16.842639 25328 tablet_bootstrap.cc:492] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489: No bootstrap required, opened a new log
I20260812 00:10:16.842693 25328 ts_tablet_manager.cc:1397] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260812 00:10:16.842733 25352 ts_tablet_manager.cc:1939] T ca6037053b3a487da67e710e12bb09e0 P 685c39362ea4458da9e963663e307314: Deleting consensus metadata
I20260812 00:10:16.842859 25328 raft_consensus.cc:348] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.842960 25328 raft_consensus.cc:374] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:16.842990 25328 raft_consensus.cc:729] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17508a2d617f4d4eb79e5f27d5447489, State: Initialized, Role: FOLLOWER
I20260812 00:10:16.843045 25328 consensus_queue.cc:260] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.843057 24775 catalog_manager.cc:4841] TS 685c39362ea4458da9e963663e307314 (127.16.99.194:37627): tablet ca6037053b3a487da67e710e12bb09e0 (table test-table [id=e323429fbb4343a3aad0e17856e7f7a4]) successfully deleted
I20260812 00:10:16.843156 25328 ts_tablet_manager.cc:1428] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:16.904496 25333 raft_consensus.cc:482] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:16.904690 25333 raft_consensus.cc:504] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.905090 25333 leader_election.cc:288] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.909142 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.905528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:37821 (local address 127.16.99.193:43335)
0812 00:10:16.905578 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.905580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.905593 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:16.905710 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.905712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.905724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.905756 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.906033 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.906773 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.908325 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.908721 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.908815 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.908825 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.909012 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.909057 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:10:16.909168 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.905264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43335 (local address 127.16.99.194:37821)
0812 00:10:16.905535 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.905546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.905607 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.905860 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.905860 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.905861 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.905941 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.905943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.906925 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.906927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.908228 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.908231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.908813 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.908815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.908901 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.908905 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.909011 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.909057 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":59,"thread_start_us":126,"threads_started":1}
I20260812 00:10:16.909386 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.905459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:40173 (local address 127.16.99.195:38133)
0812 00:10:16.905509 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.905511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.905619 (+ 108us) server_negotiation.cc:407] Connection header received
0812 00:10:16.905727 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.905729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.905740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.905824 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:16.906003 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.906791 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.908417 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.908956 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.909011 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.909020 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.909254 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.909312 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:10:16.909622 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.905367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:38133 (local address 127.16.99.194:40173)
0812 00:10:16.905548 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.905557 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.905642 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.905829 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.905830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.905833 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:16.905915 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.905917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.906943 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.906945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.908294 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.908298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.909089 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.909090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.909167 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.909171 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.909491 (+ 320us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.909539 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":80,"thread_start_us":63,"threads_started":1}
I20260812 00:10:16.910495 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8b701bf40748a69871ad07b06d8e87" candidate_uuid: "685c39362ea4458da9e963663e307314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17508a2d617f4d4eb79e5f27d5447489" is_pre_election: true
I20260812 00:10:16.910512 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8b701bf40748a69871ad07b06d8e87" candidate_uuid: "685c39362ea4458da9e963663e307314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33" is_pre_election: true
I20260812 00:10:16.910632 25250 raft_consensus.cc:2454] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c39362ea4458da9e963663e307314 in term 0.
I20260812 00:10:16.910633 24956 raft_consensus.cc:2454] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c39362ea4458da9e963663e307314 in term 0.
I20260812 00:10:16.910830 25036 leader_election.cc:302] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, 685c39362ea4458da9e963663e307314; no voters:
I20260812 00:10:16.911039 25333 raft_consensus.cc:2781] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:16.911087 25333 raft_consensus.cc:482] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:16.911110 25333 raft_consensus.cc:3037] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.912078 25333 raft_consensus.cc:504] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.912388 25333 leader_election.cc:288] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [CANDIDATE]: Term 1 election: Requested vote from peers 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335), a2e00109249946c982787368b98b2e33 (127.16.99.195:38133)
I20260812 00:10:16.912559 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8b701bf40748a69871ad07b06d8e87" candidate_uuid: "685c39362ea4458da9e963663e307314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17508a2d617f4d4eb79e5f27d5447489"
I20260812 00:10:16.912616 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8b701bf40748a69871ad07b06d8e87" candidate_uuid: "685c39362ea4458da9e963663e307314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e00109249946c982787368b98b2e33"
I20260812 00:10:16.912663 24956 raft_consensus.cc:3037] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.912689 25250 raft_consensus.cc:3037] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:16.913489 24956 raft_consensus.cc:2454] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c39362ea4458da9e963663e307314 in term 1.
I20260812 00:10:16.913487 25250 raft_consensus.cc:2454] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c39362ea4458da9e963663e307314 in term 1.
I20260812 00:10:16.913709 25036 leader_election.cc:302] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17508a2d617f4d4eb79e5f27d5447489, 685c39362ea4458da9e963663e307314; no voters:
I20260812 00:10:16.913816 25333 raft_consensus.cc:2781] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:16.913972 25333 raft_consensus.cc:686] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 1 LEADER]: Becoming Leader. State: Replica: 685c39362ea4458da9e963663e307314, State: Running, Role: LEADER
I20260812 00:10:16.914049 25333 consensus_queue.cc:237] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } }
I20260812 00:10:16.915017 24792 catalog_manager.cc:5495] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c39362ea4458da9e963663e307314 (127.16.99.194). New cstate: current_term: 1 leader_uuid: "685c39362ea4458da9e963663e307314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "685c39362ea4458da9e963663e307314" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 } health_report { overall_health: UNKNOWN } } }
I20260812 00:10:16.924764 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.921802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.16.99.194:37627)
0812 00:10:16.921854 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.921856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.921918 (+ 62us) server_negotiation.cc:407] Connection header received
0812 00:10:16.922013 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.922014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.922032 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.922102 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:16.922238 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.922792 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.923879 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.924078 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.924156 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.924380 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.924494 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.924562 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.924642 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.924662 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:10:16.931175 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.928089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:59109 (local address 127.16.99.193:43335)
0812 00:10:16.928171 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.928177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.928219 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:10:16.928290 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.928291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.928301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.928352 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:16.928546 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.929042 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.930406 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.930625 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.930694 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.930921 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.930961 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.931024 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.931086 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.931114 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:10:16.931375 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.927959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43335 (local address 127.16.99.194:59109)
0812 00:10:16.928158 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.928170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.928236 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.928356 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.928357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.928359 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:10:16.928448 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.928450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.929144 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.929145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.930072 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.930076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.930709 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.930710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.930806 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.930829 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.931027 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.931030 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.931074 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.931126 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260812 00:10:16.936230 24956 raft_consensus.cc:1261] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Refusing update from remote peer 685c39362ea4458da9e963663e307314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.936458 25250 raft_consensus.cc:1261] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Refusing update from remote peer 685c39362ea4458da9e963663e307314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:16.936535 25333 consensus_queue.cc:1035] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17508a2d617f4d4eb79e5f27d5447489" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.936692 25411 consensus_queue.cc:1035] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e00109249946c982787368b98b2e33" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:16.946089 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.943029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37627 (local address 127.16.99.193:53399)
0812 00:10:16.943083 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.943096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:16.943136 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:16.943415 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:16.943416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:16.943418 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 00:10:16.943470 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.943472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.944268 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.944270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.945019 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:16.945022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.945595 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.945596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:16.945661 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.945670 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.945887 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.945890 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:16.945936 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 00:10:16.946003 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 00:10:16.946419 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.943226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:53399 (local address 127.16.99.194:37627)
0812 00:10:16.943274 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:16.943276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:16.943284 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:16.943307 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:16.943308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:16.943317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:16.943367 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:16.943541 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.944074 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.945179 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:16.945487 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:16.945593 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:16.945762 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:16.945811 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:16.945862 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:16.946269 (+ 407us) server_negotiation.cc:299] Negotiation successful
0812 00:10:16.946303 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:10:16.957885 24792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.958033 24792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:16.958921 24792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af90fc00a76b4d2985ee272ff5c093e6: Sending DeleteTablet for 3 replicas of tablet fd8b701bf40748a69871ad07b06d8e87
I20260812 00:10:16.959278 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8b701bf40748a69871ad07b06d8e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:58970
I20260812 00:10:16.959401 25354 tablet_replica.cc:330] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489: stopping tablet replica
I20260812 00:10:16.959385 25229 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8b701bf40748a69871ad07b06d8e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:46476
I20260812 00:10:16.959456 25354 raft_consensus.cc:2229] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.959496 25354 raft_consensus.cc:2258] T fd8b701bf40748a69871ad07b06d8e87 P 17508a2d617f4d4eb79e5f27d5447489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.959573 25353 tablet_replica.cc:330] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: stopping tablet replica
I20260812 00:10:16.959589 25083 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8b701bf40748a69871ad07b06d8e87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:16 UTC) from {username='oryx', principal='oryx/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34430
I20260812 00:10:16.959631 25353 raft_consensus.cc:2229] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:16.959671 25353 raft_consensus.cc:2258] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.959688 25352 tablet_replica.cc:330] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: stopping tablet replica
I20260812 00:10:16.959740 25352 raft_consensus.cc:2229] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:16.959861 25352 raft_consensus.cc:2258] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:16.960325 25352 ts_tablet_manager.cc:1905] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.960568 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 24869
I20260812 00:10:16.963850 25352 ts_tablet_manager.cc:1918] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:10:16.963933 25352 log.cc:1196] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/fd8b701bf40748a69871ad07b06d8e87
I20260812 00:10:16.964206 25352 ts_tablet_manager.cc:1939] T fd8b701bf40748a69871ad07b06d8e87 P 685c39362ea4458da9e963663e307314: Deleting consensus metadata
I20260812 00:10:16.964648 24775 catalog_manager.cc:4841] TS 685c39362ea4458da9e963663e307314 (127.16.99.194:37627): tablet fd8b701bf40748a69871ad07b06d8e87 (table test-table [id=2502026236b84fc3bdf4464d4078e9ae]) successfully deleted
I20260812 00:10:16.966121 25353 ts_tablet_manager.cc:1905] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:16.967928 25353 ts_tablet_manager.cc:1918] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 00:10:16.968005 25353 log.cc:1196] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/fd8b701bf40748a69871ad07b06d8e87
I20260812 00:10:16.968230 25353 ts_tablet_manager.cc:1939] T fd8b701bf40748a69871ad07b06d8e87 P a2e00109249946c982787368b98b2e33: Deleting consensus metadata
I20260812 00:10:16.968865 24775 catalog_manager.cc:4841] TS a2e00109249946c982787368b98b2e33 (127.16.99.195:38133): tablet fd8b701bf40748a69871ad07b06d8e87 (table test-table [id=2502026236b84fc3bdf4464d4078e9ae]) successfully deleted
W20260812 00:10:16.974772 24778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:43335 (error 108)
I20260812 00:10:16.975236 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 25016
I20260812 00:10:16.976802 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:16.975211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43335 (local address 127.0.0.1:58994)
0812 00:10:16.976654 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:16.976739 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":54,"threads_started":1}
W20260812 00:10:16.979633 24778 catalog_manager.cc:4584] TS 17508a2d617f4d4eb79e5f27d5447489 (127.16.99.193:43335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd8b701bf40748a69871ad07b06d8e87: Network error: Client connection negotiation failed: client connection to 127.16.99.193:43335: connect: Connection refused (error 111)
I20260812 00:10:16.984798 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 25162
I20260812 00:10:16.993394 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 24759
2026-08-12T00:10:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2676 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:17 dist-test-slave-zpfg krb5kdc[25425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:17 dist-test-slave-zpfg krb5kdc[25425](info): set up 2 sockets
Aug 12 00:10:17 dist-test-slave-zpfg krb5kdc[25425](info): commencing operation
krb5kdc: starting...
W20260812 00:10:18.095986 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:18 dist-test-slave-zpfg krb5kdc[25425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:10:18.246649 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39045
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:18.339758 25440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:18.340014 25440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:18.340040 25440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:18.341172 25440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:18.341212 25440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:18.341226 25440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:18.341239 25440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:18.341251 25440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:18.342732 25440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:18.342944 25440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:18.343124 25440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:18.348713 25445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:18.348690 25448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:18.348771 25446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:18.349025 25440 server_base.cc:1034] running on GCE node
I20260812 00:10:18.349413 25440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:18.349799 25440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:18.351025 25440 hybrid_clock.cc:648] HybridClock initialized: now 1786493418351007 us; error 49 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T00:10:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1368 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:18 dist-test-slave-zpfg krb5kdc[25453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:18 dist-test-slave-zpfg krb5kdc[25453](info): set up 2 sockets
Aug 12 00:10:18 dist-test-slave-zpfg krb5kdc[25453](info): commencing operation
krb5kdc: starting...
W20260812 00:10:19.516229 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:19 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:10:19.653393 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39261
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:19.739689 25467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:19.739946 25467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:19.739985 25467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:19.741166 25467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:19.741240 25467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:19.741263 25467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:19.741284 25467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:19.741339 25467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:19.742987 25467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:39261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:19.743286 25467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:19.743533 25467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:19.749004 25472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:19.749011 25475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:19.749042 25473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:19.749284 25467 server_base.cc:1034] running on GCE node
I20260812 00:10:19.749588 25467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:19.749976 25467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:19.751166 25467 hybrid_clock.cc:648] HybridClock initialized: now 1786493419751194 us; error 83 us; skew 500 ppm
Aug 12 00:10:19 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493419, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:19.754194 25467 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:10:19.754731 25467 webserver.cc:460] Webserver started at http://127.16.99.254:44333/ using document root <none> and password file <none>
I20260812 00:10:19.754952 25467 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:19.754997 25467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:19.755100 25467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:19.756187 25467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "113127d5fe0f47108d8ef136170e3ca3"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "5d6c78cb1a96a68d7356b807899a91e8"
server_key_iv: "25c8c9cb1303d4bbdc76906cce279c47"
server_key_version: "encryptionkey@0"
I20260812 00:10:19.756512 25467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "113127d5fe0f47108d8ef136170e3ca3"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "5d6c78cb1a96a68d7356b807899a91e8"
server_key_iv: "25c8c9cb1303d4bbdc76906cce279c47"
server_key_version: "encryptionkey@0"
I20260812 00:10:19.757886 25467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:19.758898 25482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:19.759161 25467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:10:19.759238 25467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "113127d5fe0f47108d8ef136170e3ca3"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "5d6c78cb1a96a68d7356b807899a91e8"
server_key_iv: "25c8c9cb1303d4bbdc76906cce279c47"
server_key_version: "encryptionkey@0"
I20260812 00:10:19.759357 25467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:19.770181 25467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:19.772472 25467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:19.772640 25467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:19.777007 25467 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:39261
I20260812 00:10:19.777091 25535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:39261 every 8 connection(s)
I20260812 00:10:19.777434 25467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:19.778597 25536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:19.779847 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 25467
I20260812 00:10:19.779971 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:19.780164 16783 external_mini_cluster.cc:1428] Setting key 774652e130bc8ca7597c922da3b0bbc2
I20260812 00:10:19.781534 25536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Bootstrap starting.
I20260812 00:10:19.782369 25536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:19.782716 25536 log.cc:824] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:19.783469 25536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: No bootstrap required, opened a new log
I20260812 00:10:19.786012 25536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:19.786228 25536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:19.786271 25536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113127d5fe0f47108d8ef136170e3ca3, State: Initialized, Role: FOLLOWER
I20260812 00:10:19.786477 25536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:19.786586 25536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:19.786628 25536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:19.786690 25536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:19.787429 25536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:19.787530 25536 leader_election.cc:302] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 113127d5fe0f47108d8ef136170e3ca3; no voters:
Aug 12 00:10:19 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:19.787751 25536 leader_election.cc:288] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:19.787797 25541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:19.787977 25541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 LEADER]: Becoming Leader. State: Replica: 113127d5fe0f47108d8ef136170e3ca3, State: Running, Role: LEADER
I20260812 00:10:19.788077 25536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:19.788133 25541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:19.788552 25541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 113127d5fe0f47108d8ef136170e3ca3. Latest consensus state: current_term: 1 leader_uuid: "113127d5fe0f47108d8ef136170e3ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } } }
I20260812 00:10:19.788753 25541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:19.788640 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "113127d5fe0f47108d8ef136170e3ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } } }
I20260812 00:10:19.789037 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:19.789208 25550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:19.789809 25550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:19.791750 25550 catalog_manager.cc:1347] Generated new cluster ID: d1e561fcc6bb4a2ab9db5ce68aa10fbb
I20260812 00:10:19.791813 25550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:19.792577 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:19.780845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.16.99.254:39261)
0812 00:10:19.781028 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:19.781037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:19.781063 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:10:19.781504 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:19.781512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:19.781562 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:19.781737 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:19.782588 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:19.783581 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:19.784650 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:19.785109 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:19.785193 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:19.788247 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:19.788251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:19.788251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:19.788265 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:19.790637 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:19.791154 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:19.791158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:19.791158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:19.791273 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:19.791518 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:19.791519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:19.791520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:19.791900 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:19.792036 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:19.792165 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 00:10:19.792347 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260812 00:10:19.813318 25550 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:19.813989 25550 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:19.816414 25550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Generated new TSK 0
I20260812 00:10:19.816645 25550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:10:19.853085 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39261
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:19.944136 25563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:19.944458 25563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:19.944514 25563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:19.946143 25563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:19.946250 25563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:19.946307 25563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:19.948160 25563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:19.948493 25563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:19.948804 25563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:19.955009 25569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:19.955031 25572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:19.955009 25570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:19.955543 25563 server_base.cc:1034] running on GCE node
I20260812 00:10:19.955943 25563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:19.956288 25563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:19.957485 25563 hybrid_clock.cc:648] HybridClock initialized: now 1786493419957474 us; error 35 us; skew 500 ppm
Aug 12 00:10:19 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493419, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:19.960613 25563 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:10:19.961069 25563 webserver.cc:460] Webserver started at http://127.16.99.193:45257/ using document root <none> and password file <none>
I20260812 00:10:19.961283 25563 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:19.961331 25563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:19.961493 25563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:19.962822 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "24424f17cb3541d6af6f851bead0daef"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "ed003aa5b10b79627b2e4553325e046d"
server_key_iv: "34297650be9ae77574762f59253e9a9e"
server_key_version: "encryptionkey@0"
I20260812 00:10:19.963178 25563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "24424f17cb3541d6af6f851bead0daef"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "ed003aa5b10b79627b2e4553325e046d"
server_key_iv: "34297650be9ae77574762f59253e9a9e"
server_key_version: "encryptionkey@0"
I20260812 00:10:19.964725 25563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:10:19.965718 25579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:19.966065 25563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:19.966214 25563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "24424f17cb3541d6af6f851bead0daef"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "ed003aa5b10b79627b2e4553325e046d"
server_key_iv: "34297650be9ae77574762f59253e9a9e"
server_key_version: "encryptionkey@0"
I20260812 00:10:19.966270 25563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:19.993060 25563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:19.995760 25563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:19.995894 25563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:19.996126 25563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:19.996495 25563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:19.996536 25563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:19.996558 25563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:19.996603 25563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.002905 25563 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:40923
I20260812 00:10:20.003049 25696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:40923 every 8 connection(s)
I20260812 00:10:20.003394 25563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:20.010116 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 25563
I20260812 00:10:20.010242 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:20.010468 16783 external_mini_cluster.cc:1428] Setting key c72a108f9b21534851046f7918742e47
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493419, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:20.015674 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.004277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:60263 (local address 127.16.99.254:39261)
0812 00:10:20.004323 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.004325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.004556 (+ 231us) server_negotiation.cc:407] Connection header received
0812 00:10:20.005051 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.005052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.005062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.005115 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.006398 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.006986 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.008266 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.008518 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.008583 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.011797 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:20.011798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:20.011799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:20.011807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:20.013798 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:20.014328 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:20.014329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:20.014330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:20.014433 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:20.014785 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:20.014786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:20.014786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:20.014949 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:20.015026 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:20.015391 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 00:10:20.015520 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:10:20.015846 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.004239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:60263)
0812 00:10:20.004444 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.004483 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.004961 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.005310 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.005313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.005969 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.006193 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.006198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.007097 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.007098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.008095 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.008097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.008614 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.008614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.008779 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.009444 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.009453 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.011665 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:20.013921 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:20.013931 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:20.013934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:20.014235 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:20.014527 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:20.014528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:20.014528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:20.014699 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:20.015046 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:20.015054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:20.015235 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 00:10:20.015346 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 00:10:20.015534 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":139,"threads_started":1}
I20260812 00:10:20.016518 25697 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39261
I20260812 00:10:20.016670 25697 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:20.016892 25697 heartbeater.cc:511] Master 127.16.99.254:39261 requested a full tablet report, sending...
I20260812 00:10:20.017541 25500 ts_manager.cc:194] Registered new tserver with Master: 24424f17cb3541d6af6f851bead0daef (127.16.99.193:40923)
I20260812 00:10:20.018613 25500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:60263
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:10:20.043045 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39261
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:20.128818 25702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:20.129045 25702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:20.129071 25702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:20.130331 25702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:20.130409 25702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:20.130438 25702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:20.131999 25702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:20.132273 25702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:20.132539 25702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:20.138299 25707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.138299 25702 server_base.cc:1034] running on GCE node
W20260812 00:10:20.138357 25708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:20.138316 25710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.138873 25702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:20.139148 25702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:20.140309 25702 hybrid_clock.cc:648] HybridClock initialized: now 1786493420140286 us; error 41 us; skew 500 ppm
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:20.143203 25702 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:10:20.143734 25702 webserver.cc:460] Webserver started at http://127.16.99.194:41421/ using document root <none> and password file <none>
I20260812 00:10:20.143992 25702 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:20.144040 25702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:20.144179 25702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:20.145298 25702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "5f6a8b66c90841528cc50df7b3c5c8af"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "48a090c8498309969f05efe7a5f63f1a"
server_key_iv: "475ab29d9cc7a1d66f57dd78138bb371"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.145673 25702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "5f6a8b66c90841528cc50df7b3c5c8af"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "48a090c8498309969f05efe7a5f63f1a"
server_key_iv: "475ab29d9cc7a1d66f57dd78138bb371"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.147194 25702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:20.148108 25717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.148370 25702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:10:20.148480 25702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "5f6a8b66c90841528cc50df7b3c5c8af"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "48a090c8498309969f05efe7a5f63f1a"
server_key_iv: "475ab29d9cc7a1d66f57dd78138bb371"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.148543 25702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:20.171595 25702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:20.174079 25702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:20.174216 25702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:20.174500 25702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:20.174793 25702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:20.174823 25702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.174849 25702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:20.174861 25702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.181386 25702 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:45973
I20260812 00:10:20.181499 25838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:45973 every 8 connection(s)
I20260812 00:10:20.181983 25702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:20.190047 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 25702
I20260812 00:10:20.190183 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:20.190477 16783 external_mini_cluster.cc:1428] Setting key 628abae263a923bcb52fc5cd8fdc1530
I20260812 00:10:20.193264 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.183047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:43935 (local address 127.16.99.254:39261)
0812 00:10:20.183191 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.183193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.183243 (+ 50us) server_negotiation.cc:407] Connection header received
0812 00:10:20.183830 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.183832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.183844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.183907 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.184902 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.185419 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.186513 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.186725 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.186820 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.189605 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:20.189606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:20.189607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:20.189614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:20.191648 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:20.192109 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:20.192111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:20.192111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:20.192179 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:20.192476 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:20.192478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:20.192478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:20.192663 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:20.192747 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:20.193050 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 00:10:20.193155 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260812 00:10:20.193506 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.182934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:43935)
0812 00:10:20.183117 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.183163 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.183707 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.184001 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.184004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.184591 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.184797 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.184802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.185521 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.185523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.186414 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.186417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.186834 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.186836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.186970 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.187534 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.187544 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.189497 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:20.191739 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:20.191750 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:20.191752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:20.192011 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:20.192247 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:20.192248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:20.192248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:20.192380 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:20.192736 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:20.192744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:20.192909 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 00:10:20.193029 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:10:20.193233 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260812 00:10:20.194252 25839 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39261
I20260812 00:10:20.194408 25839 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:20.194741 25839 heartbeater.cc:511] Master 127.16.99.254:39261 requested a full tablet report, sending...
I20260812 00:10:20.195262 25500 ts_manager.cc:194] Registered new tserver with Master: 5f6a8b66c90841528cc50df7b3c5c8af (127.16.99.194:45973)
I20260812 00:10:20.195801 25500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:43935
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:10:20.221618 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39261
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:20.310137 25844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:20.310436 25844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:20.310482 25844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:20.311751 25844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:20.311823 25844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:20.311870 25844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:20.313419 25844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:20.313704 25844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:20.313957 25844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:20.319427 25850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.319670 25844 server_base.cc:1034] running on GCE node
W20260812 00:10:20.319413 25852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:20.319413 25849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.320037 25844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:20.320286 25844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:20.321463 25844 hybrid_clock.cc:648] HybridClock initialized: now 1786493420321438 us; error 42 us; skew 500 ppm
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:20.324115 25844 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:10:20.324545 25844 webserver.cc:460] Webserver started at http://127.16.99.195:43487/ using document root <none> and password file <none>
I20260812 00:10:20.324741 25844 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:20.324821 25844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:20.324954 25844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:20.326071 25844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "f3d7a21fc409497184eb16166b4913a7"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "2242c7c39a111b48d1ab8821062a8cf6"
server_key_iv: "095a74384a98b5c2c0ee78fd4950a276"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.326467 25844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "f3d7a21fc409497184eb16166b4913a7"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "2242c7c39a111b48d1ab8821062a8cf6"
server_key_iv: "095a74384a98b5c2c0ee78fd4950a276"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.327952 25844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:20.328707 25861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.328894 25844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:20.328996 25844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "f3d7a21fc409497184eb16166b4913a7"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "2242c7c39a111b48d1ab8821062a8cf6"
server_key_iv: "095a74384a98b5c2c0ee78fd4950a276"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.329059 25844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:20.340416 25844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:20.342568 25844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:20.342698 25844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:20.342911 25844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:20.343256 25844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:20.343308 25844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.343333 25844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:20.343345 25844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.350085 25844 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:38755
I20260812 00:10:20.350152 25980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:38755 every 8 connection(s)
I20260812 00:10:20.350572 25844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:20.357887 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 25844
I20260812 00:10:20.357965 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:20.358119 16783 external_mini_cluster.cc:1428] Setting key 0868ede9b03b3162fb81a20b2c00a6dc
I20260812 00:10:20.361243 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.351392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40579 (local address 127.16.99.254:39261)
0812 00:10:20.351440 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.351444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.351640 (+ 196us) server_negotiation.cc:407] Connection header received
0812 00:10:20.352186 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.352187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.352198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.352264 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.353351 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.353875 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.354956 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.355153 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.355213 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.357587 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:20.357589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:20.357589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:20.357597 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:20.359544 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:20.360043 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:20.360044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:20.360044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:20.360099 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:20.360554 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:20.360554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:20.360555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:20.360675 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:20.360732 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:20.361067 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 00:10:20.361165 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:10:20.361377 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.351328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:40579)
0812 00:10:20.351498 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.351544 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.352085 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.352363 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.352366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.353047 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.353253 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.353258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.353966 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.353966 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.354866 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.354869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.355235 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.355235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.355345 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.355915 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.355924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.357491 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:20.359679 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:20.359688 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:20.359690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:20.359961 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:20.360330 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:20.360331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:20.360332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:20.360475 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:20.360758 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:20.360765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:20.360905 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 00:10:20.361004 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 00:10:20.361144 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260812 00:10:20.362007 25981 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39261
I20260812 00:10:20.362139 25981 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:20.362413 25981 heartbeater.cc:511] Master 127.16.99.254:39261 requested a full tablet report, sending...
I20260812 00:10:20.362890 25500 ts_manager.cc:194] Registered new tserver with Master: f3d7a21fc409497184eb16166b4913a7 (127.16.99.195:38755)
I20260812 00:10:20.363300 25500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:40579
I20260812 00:10:20.370605 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab.
Entry for principal oryx/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab.
WARNING: no policy specified for oryx/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab.
Entry for principal oryx/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab.
WARNING: no policy specified for oryx/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab.
Entry for principal oryx/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab.
I20260812 00:10:20.458181 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 25563
I20260812 00:10:20.465132 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 25702
I20260812 00:10:20.471218 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 25844
I20260812 00:10:20.479384 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 25467
I20260812 00:10:20.487900 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39261
--webserver_interface=127.16.99.254
--webserver_port=44333
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:20.573269 25990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:20.573549 25990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:20.573593 25990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:20.574910 25990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:20.574990 25990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:20.575013 25990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:20.575033 25990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:20.575066 25990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:20.576678 25990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:39261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=44333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:20.577005 25990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:20.577260 25990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:20.582669 25995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:20.582610 25996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:20.582641 25998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.582990 25990 server_base.cc:1034] running on GCE node
I20260812 00:10:20.583253 25990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:20.583600 25990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:20.584756 25990 hybrid_clock.cc:648] HybridClock initialized: now 1786493420584748 us; error 47 us; skew 500 ppm
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493420, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:20.594592 25990 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:10:20.595198 25990 webserver.cc:460] Webserver started at http://127.16.99.254:44333/ using document root <none> and password file <none>
I20260812 00:10:20.595490 25990 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:20.595557 25990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:20.597311 25990 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:10:20.598171 26005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.598430 25990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:20.598551 25990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "113127d5fe0f47108d8ef136170e3ca3"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "5d6c78cb1a96a68d7356b807899a91e8"
server_key_iv: "25c8c9cb1303d4bbdc76906cce279c47"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.598975 25990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:20.616278 25990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:20.618845 25990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:20.619053 25990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:20.623974 25990 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:39261
I20260812 00:10:20.624030 26059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:39261 every 8 connection(s)
I20260812 00:10:20.624488 25990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:20.626204 26060 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:10:20.627019 26060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Bootstrap starting.
I20260812 00:10:20.634306 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 25990
I20260812 00:10:20.634750 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:40923
--local_ip_for_outbound_sockets=127.16.99.193
--tserver_master_addrs=127.16.99.254:39261
--webserver_port=45257
--webserver_interface=127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:10:20.636345 26060 log.cc:824] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:20.638655 26060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:10:20.638989 26060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Bootstrap complete.
I20260812 00:10:20.641472 26060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:20.641706 26060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113127d5fe0f47108d8ef136170e3ca3, State: Initialized, Role: FOLLOWER
I20260812 00:10:20.641888 26060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:20.641969 26060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:20.642035 26060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:20.642086 26060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:10:20.643245 26060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:20.643344 26060 leader_election.cc:302] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 113127d5fe0f47108d8ef136170e3ca3; no voters:
I20260812 00:10:20.643591 26060 leader_election.cc:288] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:10:20.643630 26065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:10:20.643751 26065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [term 2 LEADER]: Becoming Leader. State: Replica: 113127d5fe0f47108d8ef136170e3ca3, State: Running, Role: LEADER
I20260812 00:10:20.643911 26060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:20.643941 26065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } }
I20260812 00:10:20.644317 26066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "113127d5fe0f47108d8ef136170e3ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } } }
I20260812 00:10:20.644459 26066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:20.644315 26067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 113127d5fe0f47108d8ef136170e3ca3. Latest consensus state: current_term: 2 leader_uuid: "113127d5fe0f47108d8ef136170e3ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113127d5fe0f47108d8ef136170e3ca3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39261 } } }
I20260812 00:10:20.644909 26067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:20.645015 26072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:20.645653 26072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:20.645843 26072 catalog_manager.cc:1259] Loaded cluster ID: d1e561fcc6bb4a2ab9db5ce68aa10fbb
I20260812 00:10:20.646049 26072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:20.647673 26072 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:20.647917 26072 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 113127d5fe0f47108d8ef136170e3ca3: Loaded TSK: 0
I20260812 00:10:20.648170 26072 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:10:20.729878 26062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:20.730121 26062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:20.730147 26062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:20.731526 26062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:20.731586 26062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:20.731617 26062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:20.733296 26062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.193.keytab
--principal=oryx/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:40923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=45257
--tserver_master_addrs=127.16.99.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:20.733610 26062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:20.733892 26062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:20.739378 26093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:20.739430 26091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:20.739374 26090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.739638 26062 server_base.cc:1034] running on GCE node
I20260812 00:10:20.739979 26062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:20.740279 26062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:20.741452 26062 hybrid_clock.cc:648] HybridClock initialized: now 1786493420741433 us; error 40 us; skew 500 ppm
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493420, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:20.744426 26062 init.cc:377] Logged in from keytab as oryx/127.16.99.193@KRBTEST.COM (short username oryx)
I20260812 00:10:20.744995 26062 webserver.cc:460] Webserver started at http://127.16.99.193:45257/ using document root <none> and password file <none>
I20260812 00:10:20.745342 26062 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:20.745448 26062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:20.747032 26062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:10:20.747771 26100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.747993 26062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:20.748100 26062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "24424f17cb3541d6af6f851bead0daef"
format_stamp: "Formatted at 2026-08-12 00:10:19 on dist-test-slave-zpfg"
server_key: "ed003aa5b10b79627b2e4553325e046d"
server_key_iv: "34297650be9ae77574762f59253e9a9e"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.748545 26062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:20.757390 26062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:20.759900 26062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:20.760093 26062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:20.760417 26062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:20.760818 26062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:20.760872 26062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.760921 26062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:20.760941 26062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.768752 26062 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:40923
I20260812 00:10:20.768826 26216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:40923 every 8 connection(s)
I20260812 00:10:20.769210 26062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:20.771740 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 26062
I20260812 00:10:20.772073 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:45973
--local_ip_for_outbound_sockets=127.16.99.194
--tserver_master_addrs=127.16.99.254:39261
--webserver_port=41421
--webserver_interface=127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:20.779729 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.769959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:38115)
0812 00:10:20.770176 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.770226 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.770899 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.771342 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.771346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.772126 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.772418 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.772425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.774161 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.774162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.775152 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.775155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.775816 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.775818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.775954 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.776565 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.776574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.779507 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
W20260812 00:10:20.780297 26217 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:39261 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.99.254:39261: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
W20260812 00:10:20.787338 26221 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:20.770078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:38115 (local address 127.16.99.254:39261)
0812 00:10:20.770282 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.770298 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.770329 (+ 31us) server_negotiation.cc:407] Connection header received
0812 00:10:20.771093 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.771098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.771142 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.771260 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.772557 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.774020 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.775277 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.775706 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.775832 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.787052 (+ 11220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:38115: BlockingRecv error: recv got EOF from 127.16.99.193:38115 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:20.790669 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.780575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:55679)
0812 00:10:20.781011 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.781024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.781072 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.781303 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.781304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.781463 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.781531 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.781533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.782571 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.782573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.783382 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.783384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.783887 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.783888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.783958 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.787963 (+ 4005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.787969 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.790554 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":114,"threads_started":1}
W20260812 00:10:20.791203 26225 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:20.780807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:55679 (local address 127.16.99.254:39261)
0812 00:10:20.781171 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.781174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.781181 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:10:20.781207 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.781208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.781219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.781613 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.781656 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.782473 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.783483 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.783807 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.784117 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.790686 (+ 6569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:55679: BlockingRecv error: recv got EOF from 127.16.99.193:55679 (error 108)
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":84,"threads_started":1}
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:20.796525 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.791008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:43137)
0812 00:10:20.791067 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.791080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.791126 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.791426 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.791427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.791568 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.791608 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.791610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.792652 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.792653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.793155 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.793156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.793658 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.793659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.793704 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.794221 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.794226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.796441 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 00:10:20.796639 26221 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:20.791193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:43137 (local address 127.16.99.254:39261)
0812 00:10:20.791262 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.791265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.791276 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:20.791310 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.791311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.791328 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.791405 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.791692 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.792554 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.793243 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.793574 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.793648 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.796559 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:43137: BlockingRecv error: recv got EOF from 127.16.99.193:43137 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260812 00:10:20.796939 26217 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:10:20.879388 26223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:20.879663 26223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:20.879693 26223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:20.881616 26223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:20.881704 26223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:20.881755 26223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:20.884544 26223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.194.keytab
--principal=oryx/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:45973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=41421
--tserver_master_addrs=127.16.99.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:20.884922 26223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:20.885210 26223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:20.891901 26230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:20.891911 26231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.891976 26223 server_base.cc:1034] running on GCE node
W20260812 00:10:20.899540 26234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:20.899953 26223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:20.900353 26223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:20.901543 26223 hybrid_clock.cc:648] HybridClock initialized: now 1786493420901515 us; error 53 us; skew 500 ppm
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493420, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:20.904723 26223 init.cc:377] Logged in from keytab as oryx/127.16.99.194@KRBTEST.COM (short username oryx)
I20260812 00:10:20.905260 26223 webserver.cc:460] Webserver started at http://127.16.99.194:41421/ using document root <none> and password file <none>
I20260812 00:10:20.905524 26223 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:20.905616 26223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:20.907516 26223 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:20.908267 26242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.908504 26223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:20.908623 26223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "5f6a8b66c90841528cc50df7b3c5c8af"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "48a090c8498309969f05efe7a5f63f1a"
server_key_iv: "475ab29d9cc7a1d66f57dd78138bb371"
server_key_version: "encryptionkey@0"
I20260812 00:10:20.909055 26223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:20.919626 26223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:20.922185 26223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:20.922530 26223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:20.922830 26223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:20.923173 26223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:20.923233 26223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.923278 26223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:20.923324 26223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:20.929924 26223 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:45973
I20260812 00:10:20.929980 26361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:45973 every 8 connection(s)
I20260812 00:10:20.930410 26223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:10:20.937290 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 26223
I20260812 00:10:20.937670 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:38755
--local_ip_for_outbound_sockets=127.16.99.195
--tserver_master_addrs=127.16.99.254:39261
--webserver_port=43487
--webserver_interface=127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:10:20.940017 26221 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:20.931320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:53325 (local address 127.16.99.254:39261)
0812 00:10:20.931395 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.931397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.931660 (+ 263us) server_negotiation.cc:407] Connection header received
0812 00:10:20.932326 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.932327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.932341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.932416 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.933544 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.934057 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.935212 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.935452 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.935539 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.939959 (+ 4420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:53325: BlockingRecv error: recv got EOF from 127.16.99.194:53325 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:10:20.940084 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.931305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:53325)
0812 00:10:20.931516 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.931558 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.932102 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.932508 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.932512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.933205 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.933418 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.933423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.934179 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.934180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.935108 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.935110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.935578 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.935579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.935705 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.936675 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.936684 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.939856 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
W20260812 00:10:20.940446 26362 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:39261 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.99.254:39261: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:20.946477 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.940740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:53459)
0812 00:10:20.940832 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.940845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.940921 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.941088 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.941090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.941235 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.941291 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.941293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.941970 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.941971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.942644 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.942647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.943258 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.943259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.943334 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.944017 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.944024 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.946408 (+ 2384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 00:10:20.946900 26221 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:20.940882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:53459 (local address 127.16.99.254:39261)
0812 00:10:20.940950 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.940953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.940962 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:20.940992 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.940993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.941006 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.941083 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.941379 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.941882 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.942739 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.942969 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.943185 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.946523 (+ 3338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:53459: BlockingRecv error: recv got EOF from 127.16.99.194:53459 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:20 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:20.952059 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:20.946808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:58653)
0812 00:10:20.947022 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:20.947034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:20.947113 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:20.947299 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:20.947300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:20.947475 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:20.947528 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.947530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.948212 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.948213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.949119 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:20.949123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.949496 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.949498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:20.949569 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.950012 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:20.950017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:20.951964 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195}
W20260812 00:10:20.952116 26225 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:20.946889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:58653 (local address 127.16.99.254:39261)
0812 00:10:20.946945 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:20.946969 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:20.947085 (+ 116us) server_negotiation.cc:407] Connection header received
0812 00:10:20.947192 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:20.947193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:20.947205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:20.947260 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:20.947633 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.948126 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.949214 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:20.949417 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:20.949469 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:20.952053 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:58653: BlockingRecv error: recv got EOF from 127.16.99.194:58653 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260812 00:10:20.952209 26362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:10:21.026717 26366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:21.027014 26366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:21.027061 26366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:21.028360 26366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:21.028443 26366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:21.028496 26366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:21.030073 26366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/krb5kdc/oryx_127.16.99.195.keytab
--principal=oryx/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:38755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=43487
--tserver_master_addrs=127.16.99.254:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:21.030409 26366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:21.030700 26366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:21.036111 26371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:21.036051 26372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:21.036051 26375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:21.036290 26366 server_base.cc:1034] running on GCE node
I20260812 00:10:21.036645 26366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:21.036940 26366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:21.038129 26366 hybrid_clock.cc:648] HybridClock initialized: now 1786493421038103 us; error 36 us; skew 500 ppm
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493421, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:21.041127 26366 init.cc:377] Logged in from keytab as oryx/127.16.99.195@KRBTEST.COM (short username oryx)
I20260812 00:10:21.041596 26366 webserver.cc:460] Webserver started at http://127.16.99.195:43487/ using document root <none> and password file <none>
I20260812 00:10:21.041808 26366 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:21.041867 26366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:21.043485 26366 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:21.044363 26383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:21.044535 26366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:21.044636 26366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "f3d7a21fc409497184eb16166b4913a7"
format_stamp: "Formatted at 2026-08-12 00:10:20 on dist-test-slave-zpfg"
server_key: "2242c7c39a111b48d1ab8821062a8cf6"
server_key_iv: "095a74384a98b5c2c0ee78fd4950a276"
server_key_version: "encryptionkey@0"
I20260812 00:10:21.045212 26366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:21.059587 26366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:21.062139 26366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:21.062302 26366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:21.062738 26366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:21.063217 26366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:21.063316 26366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:21.063356 26366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:21.063386 26366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:21.069674 26366 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:38755
I20260812 00:10:21.069765 26499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:38755 every 8 connection(s)
I20260812 00:10:21.070127 26366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:21.074851 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 26366
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:21.078558 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:21.070935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:46939)
0812 00:10:21.071098 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:21.071151 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:21.071723 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:21.072001 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:21.072006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:21.072630 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:21.072851 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.072856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.073596 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.073597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.074495 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.074497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.075049 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.075050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.075212 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.075890 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:21.075899 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:21.078216 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
W20260812 00:10:21.078677 26225 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:21.070977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:46939 (local address 127.16.99.254:39261)
0812 00:10:21.071055 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:21.071057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:21.071246 (+ 189us) server_negotiation.cc:407] Connection header received
0812 00:10:21.071858 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:21.071859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:21.071875 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:21.071928 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:21.072977 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.073488 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.074589 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.074916 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.074999 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.078618 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:46939: BlockingRecv error: recv got EOF from 127.16.99.195:46939 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 00:10:21.078833 26500 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:39261 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.99.254:39261: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:21.083045 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:21.075418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.16.99.254:39261)
0812 00:10:21.075471 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:21.075474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:21.075483 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:21.075539 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:21.075539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:21.075547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:21.075597 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:21.076283 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.076868 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.077726 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.077940 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.078021 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.079569 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:21.079574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:21.079575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:21.079589 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:21.081406 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:21.081834 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:21.081837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:21.081838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:21.081946 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:21.082166 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:21.082167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:21.082167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:21.082483 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:21.082642 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:21.082763 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 00:10:21.082888 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:21.083896 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:21.079125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:56619)
0812 00:10:21.079278 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:21.079291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:21.079350 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:21.079536 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:21.079537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:21.079723 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:21.079779 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.079780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.080433 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.080434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.080946 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.080947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.081304 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.081305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.081384 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.081795 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:21.081800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:21.083812 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260812 00:10:21.083992 26225 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:21.079353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:56619 (local address 127.16.99.254:39261)
0812 00:10:21.079421 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:21.079423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:21.079437 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:10:21.079455 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:21.079455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:21.079466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:21.079531 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:21.079881 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.080346 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.081033 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.081230 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.081306 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.083905 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:56619: BlockingRecv error: recv got EOF from 127.16.99.195:56619 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:21.088557 26502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:21.084249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:32785)
0812 00:10:21.084296 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:21.084306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:21.084345 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:21.084462 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:21.084462 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:21.084594 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:21.084628 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.084629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.085262 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.085263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.085812 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.085814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.086117 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.086118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.086158 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.086706 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:21.086710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:21.088500 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 00:10:21.088714 26225 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:21.084289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:32785 (local address 127.16.99.254:39261)
0812 00:10:21.084323 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:21.084325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:21.084333 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:10:21.084402 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:21.084402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:21.084411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:21.084458 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:21.084694 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.085177 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.085886 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.086064 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.086109 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.088640 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:32785: BlockingRecv error: recv got EOF from 127.16.99.195:32785 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 00:10:21.088733 26500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:21.803279 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:21.797400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:49713)
0812 00:10:21.797556 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:21.797567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:21.797622 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:21.797874 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:21.797875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:21.798046 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:21.798097 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.798099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.798865 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.798866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.799439 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.799440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.800021 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.800022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.800081 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.800861 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:21.800866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:21.803156 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
W20260812 00:10:21.803371 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:21.797547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:49713 (local address 127.16.99.254:39261)
0812 00:10:21.797723 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:21.797725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:21.797742 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:10:21.797767 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:21.797767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:21.797777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:21.797849 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:21.798203 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.798788 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.799530 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.799903 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.799968 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.803249 (+ 3281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:49713: BlockingRecv error: recv got EOF from 127.16.99.193:49713 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:21 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:21.957821 26506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:21.952703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:51799)
0812 00:10:21.952859 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:21.952873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:21.952939 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:21.953131 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:21.953132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:21.953314 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:21.953366 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.953368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.954023 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.954024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.954640 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:21.954642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.955173 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.955174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:21.955228 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.955681 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:21.955686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:21.957719 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":134,"threads_started":1}
W20260812 00:10:21.957929 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:21.952906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:51799 (local address 127.16.99.254:39261)
0812 00:10:21.952987 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:21.952988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:21.952998 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:21.953026 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:21.953027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:21.953038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:21.953118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:21.953447 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.953951 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.954747 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:21.954953 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:21.955019 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:21.957864 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:51799: BlockingRecv error: recv got EOF from 127.16.99.194:51799 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 00:10:22 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:22 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:10:22.095351 26505 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:22.089438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58889 (local address 127.16.99.254:39261)
0812 00:10:22.089547 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:22.089549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:22.089691 (+ 142us) server_negotiation.cc:407] Connection header received
0812 00:10:22.089822 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:22.089823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:22.089838 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:22.089915 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:22.090412 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.090928 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.091732 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.091956 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.092045 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:22.095271 (+ 3226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:58889: BlockingRecv error: recv got EOF from 127.16.99.195:58889 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 00:10:22.095376 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:22.089238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:58889)
0812 00:10:22.089515 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:22.089545 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:22.089719 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:22.089923 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:22.089924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:22.090109 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:22.090261 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:22.090264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.091039 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.091041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:22.091625 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:22.091627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.092045 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.092046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:22.092113 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:22.092653 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:22.092657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:22.095270 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":165,"threads_started":1}
Aug 12 00:10:22 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:22 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:22.808751 26508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:22.803932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:35313)
0812 00:10:22.804201 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:22.804212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:22.804271 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:22.804433 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:22.804434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:22.804604 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:22.804659 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:22.804660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.805306 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.805307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:22.805861 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:22.805863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.806200 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.806201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:22.806246 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:22.806832 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:22.806836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:22.808687 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":207,"threads_started":1}
W20260812 00:10:22.808918 26509 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:22.804002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:35313 (local address 127.16.99.254:39261)
0812 00:10:22.804141 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:22.804143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:22.804270 (+ 127us) server_negotiation.cc:407] Connection header received
0812 00:10:22.804324 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:22.804324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:22.804336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:22.804402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:22.804724 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.805225 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.805936 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.806131 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.806184 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:22.808829 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:35313: BlockingRecv error: recv got EOF from 127.16.99.193:35313 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
Aug 12 00:10:22 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:22 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:22.964195 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:22.958468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:46695)
0812 00:10:22.958648 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:22.958669 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:22.958768 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:22.958988 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:22.958989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:22.959400 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:22.959484 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:22.959487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.960256 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.960257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:22.960821 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:22.960823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.961211 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.961212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:22.961267 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:22.961783 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:22.961790 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:22.964130 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":117,"threads_started":1}
W20260812 00:10:22.964392 26509 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:22.958565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:46695 (local address 127.16.99.254:39261)
0812 00:10:22.958642 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:22.958645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:22.958747 (+ 102us) server_negotiation.cc:407] Connection header received
0812 00:10:22.958854 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:22.958855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:22.958869 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:22.958958 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:22.959608 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.960152 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.960912 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:22.961102 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:22.961159 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:22.964318 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:46695: BlockingRecv error: recv got EOF from 127.16.99.194:46695 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 00:10:23 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:23 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:23.101339 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:23.095899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:39447)
0812 00:10:23.096200 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:23.096216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:23.096305 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:23.096535 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:23.096537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:23.096737 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:23.096801 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:23.096803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.097469 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.097470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:23.098038 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:23.098040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.098475 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.098476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:23.098524 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:23.099027 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:23.099032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:23.101275 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":78,"threads_started":1}
W20260812 00:10:23.101570 26509 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:23.096201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:39447 (local address 127.16.99.254:39261)
0812 00:10:23.096278 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:23.096280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:23.096290 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:23.096374 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:23.096375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:23.096387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:23.096440 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:23.096874 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.097388 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.098129 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.098330 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.098455 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:23.101494 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:39447: BlockingRecv error: recv got EOF from 127.16.99.195:39447 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 12 00:10:23 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:23 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:23.820844 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:23.809663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:38925)
0812 00:10:23.809848 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:23.809860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:23.809929 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:23.810147 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:23.810148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:23.810318 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:23.810438 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:23.810440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.811238 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.811239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:23.812193 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:23.812197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.812608 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.812609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:23.812706 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:23.818102 (+ 5396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:23.818109 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:23.820774 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":164,"threads_started":1}
W20260812 00:10:23.821058 26513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:23.809791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:38925 (local address 127.16.99.254:39261)
0812 00:10:23.809922 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:23.809924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:23.809937 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:23.809977 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:23.809978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:23.809989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:23.810054 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:23.810521 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.811128 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.812265 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.812511 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.812578 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:23.820967 (+ 8389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:38925: BlockingRecv error: recv got EOF from 127.16.99.193:38925 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
Aug 12 00:10:23 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:23 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:23.970181 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:23.964808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:56337)
0812 00:10:23.964976 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:23.964991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:23.965076 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:23.965242 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:23.965243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:23.965453 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:23.965505 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:23.965507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.966228 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.966229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:23.966799 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:23.966800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.967185 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.967186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:23.967250 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:23.967753 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:23.967758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:23.970065 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
W20260812 00:10:23.970271 26513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:23.965025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:56337 (local address 127.16.99.254:39261)
0812 00:10:23.965105 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:23.965107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:23.965117 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:23.965156 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:23.965157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:23.965167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:23.965234 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:23.965603 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.966129 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.966902 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:23.967105 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:23.967187 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:23.970199 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:56337: BlockingRecv error: recv got EOF from 127.16.99.194:56337 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 12 00:10:24 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:24 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:24.107610 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:24.102034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:33475)
0812 00:10:24.102214 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:24.102227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:24.102304 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:24.102616 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:24.102617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:24.102852 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:24.102911 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:24.102913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.103856 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.103858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:24.104589 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:24.104591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.104929 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.104930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:24.104971 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:24.105551 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:24.105557 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:24.107521 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
W20260812 00:10:24.107697 26513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:24.102096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:33475 (local address 127.16.99.254:39261)
0812 00:10:24.102172 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:24.102173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:24.102430 (+ 257us) server_negotiation.cc:407] Connection header received
0812 00:10:24.102484 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:24.102485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:24.102505 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:24.102634 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:24.102976 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.103730 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.104662 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.104860 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.104919 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:24.107632 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:33475: BlockingRecv error: recv got EOF from 127.16.99.195:33475 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 00:10:24 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:24 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:24.826311 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:24.821415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:40461)
0812 00:10:24.821558 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:24.821569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:24.821621 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:24.821848 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:24.821848 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:24.822021 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:24.822076 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:24.822077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.822787 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.822789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:24.823362 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:24.823364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.823744 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.823745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:24.823797 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:24.824378 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:24.824383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:24.826240 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260812 00:10:24.826503 26517 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:24.821505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:40461 (local address 127.16.99.254:39261)
0812 00:10:24.821670 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:24.821672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:24.821682 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:24.821716 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:24.821717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:24.821727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:24.821838 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:24.822154 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.822690 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.823462 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.823667 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.823728 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:24.826405 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:40461: BlockingRecv error: recv got EOF from 127.16.99.193:40461 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
Aug 12 00:10:24 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:24 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:24.976533 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:24.970991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:39637)
0812 00:10:24.971291 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:24.971304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:24.971371 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:24.971557 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:24.971559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:24.971739 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:24.971798 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:24.971800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.972603 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.972604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:24.973162 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:24.973164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.973607 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.973608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:24.973654 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:24.974369 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:24.974376 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:24.976469 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":75,"threads_started":1}
W20260812 00:10:24.976680 26517 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:24.971216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:39637 (local address 127.16.99.254:39261)
0812 00:10:24.971359 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:24.971362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:24.971374 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:24.971430 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:24.971431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:24.971445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:24.971506 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:24.971890 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.972510 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.973323 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:24.973527 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:24.973593 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:24.976592 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:39637: BlockingRecv error: recv got EOF from 127.16.99.194:39637 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 12 00:10:25 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:25 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:25.113502 26519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:25.108313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:38461)
0812 00:10:25.108487 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:25.108507 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:25.108569 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:25.108714 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:25.108715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:25.108909 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:25.108962 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:25.108964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.109613 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.109614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:25.110246 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:25.110248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.110778 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.110779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:25.110833 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:25.111385 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:25.111390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:25.113438 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
W20260812 00:10:25.113651 26517 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:25.108403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:38461 (local address 127.16.99.254:39261)
0812 00:10:25.108459 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:25.108461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:25.108570 (+ 109us) server_negotiation.cc:407] Connection header received
0812 00:10:25.108614 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:25.108614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:25.108632 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:25.108686 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:25.109029 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.109535 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.110447 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.110660 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.110726 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:25.113555 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:38461: BlockingRecv error: recv got EOF from 127.16.99.195:38461 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 12 00:10:25 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:25 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:25.832865 26520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:25.827035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:53283)
0812 00:10:25.827225 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:25.827241 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:25.827368 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:25.827529 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:25.827529 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:25.827699 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:25.827753 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:25.827755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.828455 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.828456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:25.829143 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:25.829147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.829721 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.829722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:25.829772 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:25.830423 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:25.830429 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:25.832757 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
W20260812 00:10:25.832945 26521 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:25.827183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:53283 (local address 127.16.99.254:39261)
0812 00:10:25.827332 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:25.827334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:25.827347 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:25.827431 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:25.827432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:25.827442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:25.827507 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:25.827829 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.828374 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.829279 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.829638 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.829711 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:25.832865 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:53283: BlockingRecv error: recv got EOF from 127.16.99.193:53283 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
Aug 12 00:10:25 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:25 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:25.981848 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:25.977101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:41903)
0812 00:10:25.977295 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:25.977310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:25.977379 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:25.977515 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:25.977516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:25.977685 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:25.977738 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:25.977740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.978432 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.978433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:25.979018 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:25.979020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.979350 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.979351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:25.979390 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:25.979899 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:25.979904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:25.981763 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
W20260812 00:10:25.982013 26521 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:25.977267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:41903 (local address 127.16.99.254:39261)
0812 00:10:25.977351 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:25.977353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:25.977365 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:25.977441 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:25.977442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:25.977452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:25.977494 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:25.977842 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.978369 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.979096 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:25.979281 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:25.979341 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:25.981929 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:41903: BlockingRecv error: recv got EOF from 127.16.99.194:41903 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 00:10:26 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:26 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:26.119550 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:26.114063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:40353)
0812 00:10:26.114276 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:26.114287 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:26.114418 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:26.114583 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:26.114584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:26.114776 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:26.114832 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:26.114834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.115483 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.115484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:26.116110 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:26.116112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.116511 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.116512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:26.116561 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:26.117041 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:26.117046 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:26.119484 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
W20260812 00:10:26.119750 26521 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:26.114166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40353 (local address 127.16.99.254:39261)
0812 00:10:26.114236 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:26.114238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:26.114377 (+ 139us) server_negotiation.cc:407] Connection header received
0812 00:10:26.114485 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:26.114486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:26.114501 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:26.114563 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:26.114907 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.115401 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.116226 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.116420 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.116493 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:26.119640 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:40353: BlockingRecv error: recv got EOF from 127.16.99.195:40353 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 00:10:26 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:26 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:26.838110 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:26.833398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:38343)
0812 00:10:26.833557 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:26.833569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:26.833629 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:26.833779 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:26.833780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:26.833959 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:26.834010 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:26.834011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.834682 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.834682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:26.835215 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:26.835216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.835577 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.835578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:26.835622 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:26.836117 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:26.836121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:26.838048 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
W20260812 00:10:26.838359 26525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:26.833490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:38343 (local address 127.16.99.254:39261)
0812 00:10:26.833637 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:26.833639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:26.833649 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:26.833681 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:26.833682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:26.833693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:26.833757 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:26.834079 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.834611 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.835303 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.835504 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.835561 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:26.838253 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:38343: BlockingRecv error: recv got EOF from 127.16.99.193:38343 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
Aug 12 00:10:26 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:26 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:26.988189 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:26.982553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:33419)
0812 00:10:26.982752 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:26.982773 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:26.982830 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:26.983283 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:26.983285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:26.983465 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:26.983518 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:26.983520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.984359 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.984360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:26.984961 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:26.984963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.985419 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.985420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:26.985470 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:26.986037 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:26.986042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:26.988125 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":137,"threads_started":1}
W20260812 00:10:26.988359 26525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:26.982661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:33419 (local address 127.16.99.254:39261)
0812 00:10:26.982987 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:26.982990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:26.983003 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:26.983036 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:26.983037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:26.983050 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:26.983237 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:26.983645 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.984244 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.985084 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:26.985333 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:26.985399 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:26.988261 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:33419: BlockingRecv error: recv got EOF from 127.16.99.194:33419 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Aug 12 00:10:27 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:27 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:27.125727 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:27.120146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:41889)
0812 00:10:27.120377 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:27.120388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:27.120453 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:27.120612 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:27.120613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:27.120793 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:27.120845 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:27.120846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.121596 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.121597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:27.122313 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:27.122317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.122819 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.122820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:27.122869 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:27.123440 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:27.123445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:27.125613 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":122,"threads_started":1}
W20260812 00:10:27.125833 26525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:27.120238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:41889 (local address 127.16.99.254:39261)
0812 00:10:27.120406 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:27.120409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:27.120460 (+ 51us) server_negotiation.cc:407] Connection header received
0812 00:10:27.120509 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:27.120510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:27.120523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:27.120588 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:27.120913 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.121476 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.122433 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.122717 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.122783 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:27.125762 (+ 2979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:41889: BlockingRecv error: recv got EOF from 127.16.99.195:41889 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 12 00:10:27 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:27 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:27.844270 26528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:27.838683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:52361)
0812 00:10:27.838887 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:27.838903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:27.838994 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:27.839178 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:27.839179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:27.839352 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:27.839407 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:27.839409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.840093 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.840094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:27.840698 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:27.840701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.841208 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.841210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:27.841262 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:27.841849 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:27.841854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:27.844100 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":105,"threads_started":1}
W20260812 00:10:27.844269 26529 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:27.838767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:52361 (local address 127.16.99.254:39261)
0812 00:10:27.838935 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:27.838937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:27.838975 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:10:27.839053 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:27.839054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:27.839068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:27.839130 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:27.839482 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.840007 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.840849 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.841053 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.841111 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:27.844209 (+ 3098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:52361: BlockingRecv error: recv got EOF from 127.16.99.193:52361 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
Aug 12 00:10:27 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:27 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:27.994412 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:27.988814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:44753)
0812 00:10:27.989045 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:27.989063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:27.989154 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:27.989338 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:27.989339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:27.989527 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:27.989577 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:27.989579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.990477 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.990478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:27.991054 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:27.991056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.991469 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.991470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:27.991543 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:27.992102 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:27.992107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:27.994288 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
W20260812 00:10:27.994590 26529 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:27.988905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:44753 (local address 127.16.99.254:39261)
0812 00:10:27.988995 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:27.988998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:27.989133 (+ 135us) server_negotiation.cc:407] Connection header received
0812 00:10:27.989235 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:27.989236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:27.989250 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:27.989332 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:27.989711 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.990390 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.991156 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:27.991373 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:27.991464 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:27.994505 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:44753: BlockingRecv error: recv got EOF from 127.16.99.194:44753 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 00:10:28 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:28 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:28.132310 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:28.126529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:38959)
0812 00:10:28.126762 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:28.126783 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:28.126897 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:28.127064 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:28.127066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:28.127346 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:28.127400 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:28.127403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.128098 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.128100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:28.128665 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:28.128668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.129036 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.129037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:28.129128 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:28.129630 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:28.129635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:28.132218 (+ 2583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
W20260812 00:10:28.132491 26529 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:28.126655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:38959 (local address 127.16.99.254:39261)
0812 00:10:28.126755 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:28.126758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:28.126882 (+ 124us) server_negotiation.cc:407] Connection header received
0812 00:10:28.126961 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:28.126962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:28.126977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:28.127054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:28.127494 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.127999 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.128761 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.128957 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.129036 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:28.132419 (+ 3383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:38959: BlockingRecv error: recv got EOF from 127.16.99.195:38959 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 00:10:28 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:28 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:28.855535 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:28.844909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:46413)
0812 00:10:28.845062 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:28.845072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:28.845123 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:28.845360 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:28.845362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:28.845541 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:28.845596 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:28.845598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.846631 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.846633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:28.847231 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:28.847234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.847698 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.847699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:28.847767 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:28.853049 (+ 5282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:28.853058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:28.855422 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
W20260812 00:10:28.855710 26533 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:28.845034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:46413 (local address 127.16.99.254:39261)
0812 00:10:28.845187 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:28.845189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:28.845201 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:28.845237 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:28.845238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:28.845249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:28.845342 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:28.845729 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.846526 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.847324 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.847547 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.847646 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:28.855525 (+ 7879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:46413: BlockingRecv error: recv got EOF from 127.16.99.193:46413 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":108,"threads_started":1}
Aug 12 00:10:28 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:28 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:29.001013 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:28.995094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:47941)
0812 00:10:28.995272 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:28.995284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:28.995357 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:28.995531 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:28.995531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:28.995712 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:28.995763 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:28.995765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.996702 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.996703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:28.997262 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:28.997263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.997710 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.997711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:28.997772 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:28.998448 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:28.998453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:29.000887 (+ 2434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":121,"threads_started":1}
W20260812 00:10:29.001055 26533 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:28.995282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:47941 (local address 127.16.99.254:39261)
0812 00:10:28.995363 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:28.995365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:28.995402 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:10:28.995441 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:28.995442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:28.995454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:28.995498 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:28.995877 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.996558 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.997361 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:28.997572 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:28.997704 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:29.000986 (+ 3282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:47941: BlockingRecv error: recv got EOF from 127.16.99.194:47941 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 00:10:29 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:29 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:29.138906 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:29.132995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:46641)
0812 00:10:29.133238 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:29.133253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:29.133369 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:29.133558 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:29.133559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:29.133785 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:29.133841 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:29.133843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.134678 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.134680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:29.135279 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:29.135280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.135678 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.135679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:29.135748 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:29.136260 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:29.136266 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:29.138829 (+ 2563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
W20260812 00:10:29.139073 26533 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:29.133060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:46641 (local address 127.16.99.254:39261)
0812 00:10:29.133124 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:29.133126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:29.133354 (+ 228us) server_negotiation.cc:407] Connection header received
0812 00:10:29.133437 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:29.133438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:29.133457 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:29.133551 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:29.133935 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.134570 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.135374 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.135591 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.135679 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:29.138978 (+ 3299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:46641: BlockingRecv error: recv got EOF from 127.16.99.195:46641 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 00:10:29 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:29 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:29.861428 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:29.856160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:40413)
0812 00:10:29.856392 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:29.856407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:29.856493 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:29.856696 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:29.856697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:29.856894 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:29.856947 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:29.856949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.857746 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.857747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:29.858325 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:29.858327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.858719 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.858720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:29.858764 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:29.859356 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:29.859361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:29.861363 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
W20260812 00:10:29.861677 26537 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:29.856272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:40413 (local address 127.16.99.254:39261)
0812 00:10:29.856485 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:29.856487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:29.856498 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:29.856548 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:29.856548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:29.856559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:29.856628 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:29.857045 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.857662 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.858435 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:29.858653 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:29.858710 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:29.861567 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:40413: BlockingRecv error: recv got EOF from 127.16.99.193:40413 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
Aug 12 00:10:30 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:30 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:30.007460 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:30.001814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:48011)
0812 00:10:30.002052 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:30.002068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:30.002143 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:30.002409 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:30.002410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:30.002595 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:30.002650 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:30.002659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.003426 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.003427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:30.003985 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:30.003987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.004416 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.004417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:30.004472 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:30.005021 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:30.005026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:30.007395 (+ 2369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":139,"threads_started":1}
W20260812 00:10:30.007609 26537 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:30.001901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:48011 (local address 127.16.99.254:39261)
0812 00:10:30.001982 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:30.001986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:30.002143 (+ 157us) server_negotiation.cc:407] Connection header received
0812 00:10:30.002213 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:30.002214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:30.002228 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:30.002276 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:30.002733 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.003296 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.004082 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.004316 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.004387 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:30.007521 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:48011: BlockingRecv error: recv got EOF from 127.16.99.194:48011 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 00:10:30 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:30 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:30.145670 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:30.139590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:43313)
0812 00:10:30.139804 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:30.139820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:30.139884 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:30.140073 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:30.140075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:30.140257 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:30.140311 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:30.140313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.141002 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.141003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:30.141673 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:30.141676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.142238 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.142239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:30.142295 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:30.142860 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:30.142866 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:30.145606 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
W20260812 00:10:30.145841 26537 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:30.139705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:43313 (local address 127.16.99.254:39261)
0812 00:10:30.139797 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:30.139799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:30.139886 (+ 87us) server_negotiation.cc:407] Connection header received
0812 00:10:30.139950 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:30.139950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:30.139963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:30.140010 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:30.140388 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.140904 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.141773 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.142003 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.142071 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:30.145763 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:43313: BlockingRecv error: recv got EOF from 127.16.99.195:43313 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 00:10:30 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:30 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:30.867628 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:30.862090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:48727)
0812 00:10:30.862317 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:30.862331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:30.862424 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:30.862636 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:30.862637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:30.862841 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:30.862893 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:30.862895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.863594 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.863595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:30.864246 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:30.864248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.864645 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.864646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:30.864699 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:30.865363 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:30.865370 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:30.867514 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":111,"threads_started":1}
W20260812 00:10:30.867698 26541 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:30.862224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:48727 (local address 127.16.99.254:39261)
0812 00:10:30.862412 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:30.862414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:30.862426 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:30.862482 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:30.862482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:30.862494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:30.862558 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:30.862980 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.863519 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.864355 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:30.864552 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:30.864610 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:30.867617 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:48727: BlockingRecv error: recv got EOF from 127.16.99.193:48727 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
Aug 12 00:10:31 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:31 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:31.013350 26542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:31.008101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:42367)
0812 00:10:31.008296 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:31.008312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:31.008409 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:31.008664 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:31.008665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:31.008855 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:31.008906 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:31.008908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.009618 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.009620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:31.010177 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:31.010178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.010632 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.010634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:31.010705 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:31.011253 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:31.011258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:31.013236 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
W20260812 00:10:31.013406 26541 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:31.008354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:42367 (local address 127.16.99.254:39261)
0812 00:10:31.008430 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:31.008432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:31.008444 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:31.008515 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:31.008516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:31.008531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:31.008627 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:31.008994 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.009521 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.010266 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.010544 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.010632 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:31.013337 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:42367: BlockingRecv error: recv got EOF from 127.16.99.194:42367 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 00:10:31 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:31 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:31.151327 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:31.146218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:40505)
0812 00:10:31.146410 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:31.146424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:31.146486 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:31.146682 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:31.146683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:31.146885 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:31.146952 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:31.146954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.147623 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.147624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:31.148166 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:31.148167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.148579 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.148580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:31.148649 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:31.149172 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:31.149177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:31.151205 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
W20260812 00:10:31.151367 26541 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:31.146468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40505 (local address 127.16.99.254:39261)
0812 00:10:31.146550 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:31.146552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:31.146561 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:31.146594 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:31.146594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:31.146606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:31.146673 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:31.147038 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.147539 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.148254 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.148450 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.148576 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:31.151298 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:40505: BlockingRecv error: recv got EOF from 127.16.99.195:40505 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 12 00:10:31 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:31 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:31.874086 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:31.868338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:35079)
0812 00:10:31.868521 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:31.868533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:31.868641 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:31.868902 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:31.868904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:31.869120 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:31.869193 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:31.869195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.869930 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.869931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:31.870792 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:31.870795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.871255 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.871257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:31.871309 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:31.871787 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:31.871793 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:31.874004 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
W20260812 00:10:31.874287 26545 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:31.868462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:35079 (local address 127.16.99.254:39261)
0812 00:10:31.868595 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:31.868597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:31.868609 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:10:31.868754 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:31.868756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:31.868769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:31.868850 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:31.869298 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.869824 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.870885 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:31.871084 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:31.871141 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:31.874203 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:35079: BlockingRecv error: recv got EOF from 127.16.99.193:35079 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
Aug 12 00:10:32 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:32 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:32.019915 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:32.013934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:50215)
0812 00:10:32.014130 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:32.014145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:32.014212 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:32.014507 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:32.014509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:32.014707 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:32.014762 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:32.014764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.015626 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.015627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:32.016289 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:32.016292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.016749 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.016750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:32.016799 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:32.017477 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:32.017482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:32.019839 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":134,"threads_started":1}
W20260812 00:10:32.020090 26545 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:32.014066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:50215 (local address 127.16.99.254:39261)
0812 00:10:32.014263 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:32.014265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:32.014275 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:32.014307 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:32.014308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:32.014321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:32.014452 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:32.014856 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.015519 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.016388 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.016610 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.016691 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:32.019974 (+ 3283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:50215: BlockingRecv error: recv got EOF from 127.16.99.194:50215 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Aug 12 00:10:32 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:32 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:32.157433 26547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:32.151967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:38715)
0812 00:10:32.152121 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:32.152132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:32.152218 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:32.152478 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:32.152479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:32.152724 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:32.152834 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:32.152837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.153538 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.153540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:32.154122 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:32.154123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.154574 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.154575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:32.154636 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:32.155188 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:32.155194 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:32.157359 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
W20260812 00:10:32.157559 26545 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:32.152003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:38715 (local address 127.16.99.254:39261)
0812 00:10:32.152077 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:32.152080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:32.152201 (+ 121us) server_negotiation.cc:407] Connection header received
0812 00:10:32.152343 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:32.152344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:32.152360 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:32.152425 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:32.152933 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.153448 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.154208 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.154489 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.154562 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:32.157476 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:38715: BlockingRecv error: recv got EOF from 127.16.99.195:38715 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 00:10:32 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:32 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:32.879789 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:32.874712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:42865)
0812 00:10:32.874915 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:32.874928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:32.874992 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:32.875161 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:32.875162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:32.875366 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:32.875418 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:32.875420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.876151 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.876152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:32.876749 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:32.876751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.877119 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.877120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:32.877166 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:32.877657 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:32.877662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:32.879723 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":109,"threads_started":1}
W20260812 00:10:32.879987 26549 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:32.874866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:42865 (local address 127.16.99.254:39261)
0812 00:10:32.875017 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:32.875019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:32.875028 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:32.875055 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:32.875055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:32.875067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:32.875130 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:32.875518 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.876026 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.876845 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:32.877045 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:32.877102 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:32.879893 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:42865: BlockingRecv error: recv got EOF from 127.16.99.193:42865 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
Aug 12 00:10:33 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:33 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:33.026031 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:33.020621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:35179)
0812 00:10:33.020810 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:33.020827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:33.020892 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:33.021117 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:33.021119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:33.021311 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:33.021360 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:33.021361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.022123 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.022125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:33.022801 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:33.022805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.023226 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.023227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:33.023325 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:33.023819 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:33.023824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:33.025963 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
W20260812 00:10:33.026147 26549 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:33.020766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:35179 (local address 127.16.99.254:39261)
0812 00:10:33.020836 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:33.020840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:33.020879 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:10:33.020959 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:33.020960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:33.020971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:33.021036 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:33.021461 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.022017 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.022919 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.023143 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.023214 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:33.026093 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:35179: BlockingRecv error: recv got EOF from 127.16.99.194:35179 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 12 00:10:33 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:33 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:33.163766 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:33.158043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:57959)
0812 00:10:33.158262 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:33.158273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:33.158390 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:33.158579 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:33.158580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:33.158757 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:33.158808 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:33.158810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.159548 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.159549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:33.160137 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:33.160141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.160558 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.160559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:33.160656 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:33.161290 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:33.161295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:33.163677 (+ 2382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
W20260812 00:10:33.163915 26549 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:33.158169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:57959 (local address 127.16.99.254:39261)
0812 00:10:33.158225 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:33.158227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:33.158393 (+ 166us) server_negotiation.cc:407] Connection header received
0812 00:10:33.158465 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:33.158465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:33.158478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:33.158541 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:33.158883 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.159447 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.160245 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.160479 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.160542 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:33.163831 (+ 3289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:57959: BlockingRecv error: recv got EOF from 127.16.99.195:57959 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 12 00:10:33 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:33 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:33.892664 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:33.880467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:39705)
0812 00:10:33.880660 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:33.880672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:33.880734 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:33.880952 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:33.880954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:33.881283 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:33.881338 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:33.881340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.882229 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.882231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:33.882950 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:33.882953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.883430 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.883431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:33.883498 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:33.890622 (+ 7124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:33.890629 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:33.892587 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":122,"threads_started":1}
W20260812 00:10:33.892855 26553 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:33.880599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:39705 (local address 127.16.99.254:39261)
0812 00:10:33.880750 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:33.880752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:33.880762 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:33.880821 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:33.880821 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:33.880833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:33.880897 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:33.881514 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.882119 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.883061 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:33.883308 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:33.883427 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:33.892752 (+ 9325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:39705: BlockingRecv error: recv got EOF from 127.16.99.193:39705 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
Aug 12 00:10:34 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:34 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:34.032509 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:34.026846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:58359)
0812 00:10:34.027095 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:34.027113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:34.027203 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:34.027417 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:34.027418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:34.027621 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:34.027672 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:34.027674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.028378 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.028379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:34.029012 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:34.029014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.029567 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.029569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:34.029649 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:34.030321 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:34.030326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:34.032433 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
W20260812 00:10:34.032689 26553 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:34.027086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:58359 (local address 127.16.99.254:39261)
0812 00:10:34.027201 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:34.027203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:34.027216 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:10:34.027273 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:34.027274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:34.027287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:34.027386 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:34.027754 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.028262 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.029105 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.029457 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.029527 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:34.032596 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:58359: BlockingRecv error: recv got EOF from 127.16.99.194:58359 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 00:10:34 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:34 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:34.169795 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:34.164294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:33391)
0812 00:10:34.164500 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:34.164517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:34.164585 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:34.164766 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:34.164766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:34.164981 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:34.165030 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:34.165032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.165775 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.165777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:34.166498 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:34.166501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.166925 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.166926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:34.166977 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:34.167499 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:34.167503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:34.169716 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":150,"threads_started":1}
W20260812 00:10:34.169883 26553 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:34.164380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:33391 (local address 127.16.99.254:39261)
0812 00:10:34.164471 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:34.164476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:34.164638 (+ 162us) server_negotiation.cc:407] Connection header received
0812 00:10:34.164670 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:34.164671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:34.164687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:34.164753 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:34.165100 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.165671 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.166591 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.166827 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.166894 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:34.169831 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:33391: BlockingRecv error: recv got EOF from 127.16.99.195:33391 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 00:10:34 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:34 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 00:10:34.898806 26558 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:34.893431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:52501 (local address 127.16.99.254:39261)
0812 00:10:34.893598 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:34.893600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:34.893611 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:10:34.893638 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:34.893638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:34.893649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:34.893710 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:34.894076 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.894655 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.895712 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.895932 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.896000 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:34.898741 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:52501: BlockingRecv error: recv got EOF from 127.16.99.193:52501 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260812 00:10:34.898800 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:34.893302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:52501)
0812 00:10:34.893504 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:34.893516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:34.893582 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:34.893733 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:34.893734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:34.893947 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:34.894000 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:34.894001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.894739 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.894740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:34.895609 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:34.895612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:34.896010 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:34.896011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:34.896052 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:34.896552 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:34.896556 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:34.898649 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":117,"threads_started":1}
Aug 12 00:10:35 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:35 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:35.038475 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:35.033130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:46521)
0812 00:10:35.033267 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:35.033288 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:35.033356 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:35.033532 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:35.033533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:35.033712 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:35.033763 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:35.033765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.034532 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.034533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:35.035102 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:35.035104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.035559 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.035560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:35.035633 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:35.036235 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:35.036240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:35.038333 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
W20260812 00:10:35.038566 26558 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:35.033260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:46521 (local address 127.16.99.254:39261)
0812 00:10:35.033362 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:35.033369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:35.033408 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:10:35.033439 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:35.033440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:35.033452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:35.033504 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:35.033851 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.034447 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.035250 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.035478 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.035546 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:35.038482 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:46521: BlockingRecv error: recv got EOF from 127.16.99.194:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 00:10:35 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:35 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.195@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:35.176175 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:35.170641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.195:53345)
0812 00:10:35.170885 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:35.170899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:35.171012 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:35.171319 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:35.171320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:35.171531 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:35.171590 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:35.171592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.172430 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.172431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:35.173184 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:35.173187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.173537 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.173538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:35.173583 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:35.174097 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:35.174103 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:35.176108 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":107,"threads_started":1}
W20260812 00:10:35.176421 26558 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:35.170733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:53345 (local address 127.16.99.254:39261)
0812 00:10:35.170834 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:35.170837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:35.170987 (+ 150us) server_negotiation.cc:407] Connection header received
0812 00:10:35.171205 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:35.171206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:35.171221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:35.171306 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:35.171686 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.172308 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.173250 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.173454 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.173519 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:35.176350 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.195:53345: BlockingRecv error: recv got EOF from 127.16.99.195:53345 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 00:10:35 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:35 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.193@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:35.904686 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:35.899441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.193:42197)
0812 00:10:35.899621 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:35.899634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:35.899701 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:35.899843 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:35.899843 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:35.900016 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:35.900068 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:35.900070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.900747 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.900748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:35.901403 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:35.901406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.901906 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.901908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:35.901972 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:35.902577 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:35.902583 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:35.904608 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
W20260812 00:10:35.904809 26562 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:35.899551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:42197 (local address 127.16.99.254:39261)
0812 00:10:35.899679 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:35.899681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:35.899695 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:10:35.899763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:35.899763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:35.899774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:35.899837 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:35.900131 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.900649 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.901514 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:35.901770 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:35.901884 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:35.904720 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.193:42197: BlockingRecv error: recv got EOF from 127.16.99.193:42197 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
Aug 12 00:10:36 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 00:10:36 dist-test-slave-zpfg krb5kdc[25453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.99.194@KRBTEST.COM for oryx/127.16.99.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 00:10:36.045030 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.039273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39261 (local address 127.16.99.194:46771)
0812 00:10:36.039468 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:36.039516 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:36.039620 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:36.039833 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:36.039834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:36.040024 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:36.040075 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.040076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.040796 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.040797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.041488 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.041491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.041882 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.041883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.041946 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.042512 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:36.042521 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:36.044919 (+ 2398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.99.254:39261: Server oryx/127.16.99.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
W20260812 00:10:36.045087 26562 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:36.039447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:46771 (local address 127.16.99.254:39261)
0812 00:10:36.039553 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:36.039555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:36.039596 (+ 41us) server_negotiation.cc:407] Connection header received
0812 00:10:36.039717 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:36.039718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:36.039736 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:36.039813 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:36.040164 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.040693 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.041583 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.041799 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.041882 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.045023 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.99.194:46771: BlockingRecv error: recv got EOF from 127.16.99.194:46771 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 00:10:36.085597 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 26062
I20260812 00:10:36.092144 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 26223
I20260812 00:10:36.098922 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 26366
I20260812 00:10:36.106630 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 25990
2026-08-12T00:10:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17744 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T00:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:36Z Disabled control of system clock
I20260812 00:10:36.141158 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:43157
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 00:10:36.231596 26567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:36.231833 26567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:36.231858 26567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:36.233089 26567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:36.233134 26567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:36.233147 26567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:36.233162 26567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:36.233175 26567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:36.234643 26567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:43157
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:36.234874 26567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:36.235145 26567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:36.240945 26575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.240945 26573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:36.241074 26567 server_base.cc:1034] running on GCE node
W20260812 00:10:36.241043 26572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:36.241591 26567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:36.241962 26567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:36.243156 26567 hybrid_clock.cc:648] HybridClock initialized: now 1786493436243135 us; error 49 us; skew 500 ppm
I20260812 00:10:36.244578 26567 webserver.cc:460] Webserver started at http://127.16.99.254:43403/ using document root <none> and password file <none>
I20260812 00:10:36.244901 26567 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:36.244983 26567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:36.245141 26567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:36.246282 26567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "1a05a7d088a244029b4b8d407bee335d"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "abf69c6295937f6a014265d94603b356"
server_key_iv: "47044dfa0d35b1b69d855699b15c08c2"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.246748 26567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "1a05a7d088a244029b4b8d407bee335d"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "abf69c6295937f6a014265d94603b356"
server_key_iv: "47044dfa0d35b1b69d855699b15c08c2"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.248136 26567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:36.248970 26582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.249159 26567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:36.249270 26567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "1a05a7d088a244029b4b8d407bee335d"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "abf69c6295937f6a014265d94603b356"
server_key_iv: "47044dfa0d35b1b69d855699b15c08c2"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.249351 26567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:36.258659 26567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:36.258989 26567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:36.259150 26567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:36.264448 26567 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:43157
I20260812 00:10:36.264499 26637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:43157 every 8 connection(s)
I20260812 00:10:36.264887 26567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:36.265630 26638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:36.267104 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 26567
I20260812 00:10:36.267216 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:36.267408 16783 external_mini_cluster.cc:1428] Setting key 81dcb648bfb955402b684ff36c29997c
I20260812 00:10:36.269750 26638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d: Bootstrap starting.
I20260812 00:10:36.270562 26638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:36.270913 26638 log.cc:824] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:36.271752 26638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d: No bootstrap required, opened a new log
I20260812 00:10:36.273818 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.268022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.16.99.254:43157)
0812 00:10:36.268238 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:36.268246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:36.268273 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:10:36.268718 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:36.268725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:36.268798 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:36.268916 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:36.269200 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.270193 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.271479 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.272026 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.272167 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.272976 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:36.272980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:36.272980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:36.273151 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:36.273185 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:36.273196 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:36.273247 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:36.273354 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 00:10:36.273418 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260812 00:10:36.273768 26638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a05a7d088a244029b4b8d407bee335d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 43157 } }
I20260812 00:10:36.273927 26638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:36.273963 26638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a05a7d088a244029b4b8d407bee335d, State: Initialized, Role: FOLLOWER
I20260812 00:10:36.274215 26638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a05a7d088a244029b4b8d407bee335d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 43157 } }
I20260812 00:10:36.274329 26638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:36.274392 26638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:36.274451 26638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:36.275030 26638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a05a7d088a244029b4b8d407bee335d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 43157 } }
I20260812 00:10:36.275170 26638 leader_election.cc:302] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a05a7d088a244029b4b8d407bee335d; no voters:
I20260812 00:10:36.275368 26638 leader_election.cc:288] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:36.275429 26644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:36.275665 26638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:36.275676 26644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [term 1 LEADER]: Becoming Leader. State: Replica: 1a05a7d088a244029b4b8d407bee335d, State: Running, Role: LEADER
I20260812 00:10:36.275802 26644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a05a7d088a244029b4b8d407bee335d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 43157 } }
I20260812 00:10:36.276254 26646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a05a7d088a244029b4b8d407bee335d. Latest consensus state: current_term: 1 leader_uuid: "1a05a7d088a244029b4b8d407bee335d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a05a7d088a244029b4b8d407bee335d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 43157 } } }
I20260812 00:10:36.276352 26646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:36.276259 26645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a05a7d088a244029b4b8d407bee335d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a05a7d088a244029b4b8d407bee335d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 43157 } } }
I20260812 00:10:36.276568 26645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:36.276937 26653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:36.277429 26653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:36.279340 26653 catalog_manager.cc:1347] Generated new cluster ID: 8cf7b95ebe2f45459386a46598905db2
I20260812 00:10:36.279407 26653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:36.296501 26653 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:36.297169 26653 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:36.306419 26653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a05a7d088a244029b4b8d407bee335d: Generated new TSK 0
I20260812 00:10:36.306639 26653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:10:36.310461 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:43157
--builtin_ntp_servers=127.16.99.212:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 00:10:36.397609 26664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:36.397857 26664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:36.397883 26664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:36.399235 26664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:36.399287 26664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:36.399315 26664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:36.400753 26664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:36.401024 26664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:36.401382 26664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:36.408327 26672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.408334 26670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.408334 26669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:36.408740 26664 server_base.cc:1034] running on GCE node
I20260812 00:10:36.409018 26664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:36.409355 26664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:36.410521 26664 hybrid_clock.cc:648] HybridClock initialized: now 1786493436410501 us; error 37 us; skew 500 ppm
I20260812 00:10:36.412073 26664 webserver.cc:460] Webserver started at http://127.16.99.193:38445/ using document root <none> and password file <none>
I20260812 00:10:36.412362 26664 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:36.412421 26664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:36.412559 26664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:36.413703 26664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "e06fb07a06a340299e1e4ee1302f95d3"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "eecdef633a54d8a4854547bd45864c80"
server_key_iv: "caa3d85c4585c596ed506cacf32b7f44"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.414098 26664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "e06fb07a06a340299e1e4ee1302f95d3"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "eecdef633a54d8a4854547bd45864c80"
server_key_iv: "caa3d85c4585c596ed506cacf32b7f44"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.415647 26664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:36.416608 26678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.416844 26664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:36.416957 26664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "e06fb07a06a340299e1e4ee1302f95d3"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "eecdef633a54d8a4854547bd45864c80"
server_key_iv: "caa3d85c4585c596ed506cacf32b7f44"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.417042 26664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:36.426810 26664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:36.427170 26664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:36.427412 26664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:36.427681 26664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:36.428053 26664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:36.428089 26664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.428114 26664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:36.428162 26664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.434719 26664 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:41855
I20260812 00:10:36.434825 26801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:41855 every 8 connection(s)
I20260812 00:10:36.435165 26664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:36.436784 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 26664
I20260812 00:10:36.436868 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:36.437059 16783 external_mini_cluster.cc:1428] Setting key c4e7c549107ef28eaf6f6d976fac66aa
I20260812 00:10:36.438241 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:43157
--builtin_ntp_servers=127.16.99.212:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 00:10:36.443312 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.436091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:46581 (local address 127.16.99.254:43157)
0812 00:10:36.436191 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:36.436193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:36.436617 (+ 424us) server_negotiation.cc:407] Connection header received
0812 00:10:36.437834 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:36.437835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:36.437847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:36.437902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:36.438893 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.439519 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.441333 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.441686 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.441770 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.442913 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:36.442914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:36.442915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:36.442922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:36.442944 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:36.442950 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:36.443021 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:36.443124 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 00:10:36.443171 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 00:10:36.443715 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.436030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:43157 (local address 127.16.99.193:46581)
0812 00:10:36.436291 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:36.436346 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:36.437454 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:36.438050 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:36.438055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:36.438064 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:36.438771 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.438778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.439644 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.439645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.441214 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.441218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.441802 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.441803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.441944 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.442732 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:36.442732 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:36.442765 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:36.442793 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:36.443028 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:36.443034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:36.443036 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:10:36.443099 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 00:10:36.443442 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":130,"threads_started":1}
I20260812 00:10:36.444307 26802 heartbeater.cc:347] Connected to a master server at 127.16.99.254:43157
I20260812 00:10:36.444453 26802 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:36.444715 26802 heartbeater.cc:511] Master 127.16.99.254:43157 requested a full tablet report, sending...
I20260812 00:10:36.445463 26592 ts_manager.cc:194] Registered new tserver with Master: e06fb07a06a340299e1e4ee1302f95d3 (127.16.99.193:41855)
I20260812 00:10:36.446174 26592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.193:46581
W20260812 00:10:36.530431 26805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:36.530660 26805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:36.530683 26805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:36.532105 26805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:36.532171 26805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:36.532200 26805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:36.533677 26805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:36.533984 26805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:36.534451 26805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:36.540079 26811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.540081 26813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:36.540099 26805 server_base.cc:1034] running on GCE node
W20260812 00:10:36.540082 26810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:36.540606 26805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:36.540877 26805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:36.542173 26805 hybrid_clock.cc:648] HybridClock initialized: now 1786493436542120 us; error 48 us; skew 500 ppm
I20260812 00:10:36.543936 26805 webserver.cc:460] Webserver started at http://127.16.99.194:46609/ using document root <none> and password file <none>
I20260812 00:10:36.544268 26805 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:36.544381 26805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:36.544543 26805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:36.545861 26805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "f844b299ccb545159b43d5d147493853"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "5e80db541184e8fc7bf6ca25f8f3c420"
server_key_iv: "101ee80587c50bf3dde7fbe8dca93e6b"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.546283 26805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "f844b299ccb545159b43d5d147493853"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "5e80db541184e8fc7bf6ca25f8f3c420"
server_key_iv: "101ee80587c50bf3dde7fbe8dca93e6b"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.547852 26805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:36.548772 26820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.548969 26805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:36.549091 26805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "f844b299ccb545159b43d5d147493853"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "5e80db541184e8fc7bf6ca25f8f3c420"
server_key_iv: "101ee80587c50bf3dde7fbe8dca93e6b"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.549181 26805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:36.571790 26805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:36.572184 26805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:36.572362 26805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:36.572657 26805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:36.573050 26805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:36.573108 26805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.573167 26805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:36.573195 26805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.580192 26805 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:41155
I20260812 00:10:36.580240 26939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:41155 every 8 connection(s)
I20260812 00:10:36.580639 26805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:10:36.586151 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 26805
I20260812 00:10:36.586400 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:36.586668 16783 external_mini_cluster.cc:1428] Setting key 74aaf17e3baec2d651dce00fd2d9ee0a
I20260812 00:10:36.586833 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.581579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:37475 (local address 127.16.99.254:43157)
0812 00:10:36.581660 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:36.581662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:36.581943 (+ 281us) server_negotiation.cc:407] Connection header received
0812 00:10:36.582635 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:36.582636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:36.582648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:36.582701 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:36.583459 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.583948 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.585115 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.585316 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.585376 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.586401 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:36.586403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:36.586403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:36.586410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:36.586429 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:36.586434 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:36.586514 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:36.586640 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 00:10:36.586675 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:10:36.587097 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.581546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:43157 (local address 127.16.99.194:37475)
0812 00:10:36.581806 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:36.581847 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:36.582518 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:36.582771 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:36.582775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:36.582787 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:36.583370 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.583376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.584138 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.584142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.585011 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.585014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.585403 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.585404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.585526 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.586131 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:36.586131 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:36.586177 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:36.586210 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:36.586519 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:36.586527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:36.586531 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:36.586626 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 00:10:36.586762 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":110,"threads_started":1}
I20260812 00:10:36.587680 26940 heartbeater.cc:347] Connected to a master server at 127.16.99.254:43157
I20260812 00:10:36.587805 26940 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:36.587917 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:43157
--builtin_ntp_servers=127.16.99.212:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 00:10:36.588065 26940 heartbeater.cc:511] Master 127.16.99.254:43157 requested a full tablet report, sending...
I20260812 00:10:36.588567 26592 ts_manager.cc:194] Registered new tserver with Master: f844b299ccb545159b43d5d147493853 (127.16.99.194:41155)
I20260812 00:10:36.588987 26592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.194:37475
W20260812 00:10:36.684430 26943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:36.684696 26943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:36.684736 26943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:36.686108 26943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:36.686185 26943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:36.686232 26943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:36.687793 26943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:43157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:36.688134 26943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:36.688489 26943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:36.694571 26952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.694574 26950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.694571 26949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:36.694932 26943 server_base.cc:1034] running on GCE node
I20260812 00:10:36.695130 26943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:36.695401 26943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:36.696560 26943 hybrid_clock.cc:648] HybridClock initialized: now 1786493436696546 us; error 30 us; skew 500 ppm
I20260812 00:10:36.697979 26943 webserver.cc:460] Webserver started at http://127.16.99.195:43427/ using document root <none> and password file <none>
I20260812 00:10:36.698256 26943 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:36.698429 26943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:36.698593 26943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:36.699733 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "abef6036b02c48fca9807fdfad6f1199"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "6e4ca38ab3336b72f0afb948caf8f295"
server_key_iv: "e772d1d7b473fb34ec4c2a31cf658330"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.700117 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "abef6036b02c48fca9807fdfad6f1199"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "6e4ca38ab3336b72f0afb948caf8f295"
server_key_iv: "e772d1d7b473fb34ec4c2a31cf658330"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.701541 26943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:36.702507 26958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.702692 26943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:36.702790 26943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "abef6036b02c48fca9807fdfad6f1199"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "6e4ca38ab3336b72f0afb948caf8f295"
server_key_iv: "e772d1d7b473fb34ec4c2a31cf658330"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.702876 26943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:36.726020 26943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:36.726507 26943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:36.726740 26943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:36.727022 26943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:36.727425 26943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:36.727483 26943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.727533 26943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:36.727564 26943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.734536 26943 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:40279
I20260812 00:10:36.734634 27081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:40279 every 8 connection(s)
I20260812 00:10:36.735041 26943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:36.741127 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.736011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:52959 (local address 127.16.99.254:43157)
0812 00:10:36.736108 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:36.736110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:36.736454 (+ 344us) server_negotiation.cc:407] Connection header received
0812 00:10:36.736948 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:36.736949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:36.736969 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:36.737038 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:36.737861 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.738453 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.739450 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.739733 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.739851 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.740880 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:36.740881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:36.740881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:36.740886 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:36.740899 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:36.740903 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:36.740953 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:36.741030 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 00:10:36.741056 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 00:10:36.741384 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.735992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:43157 (local address 127.16.99.195:52959)
0812 00:10:36.736266 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:36.736361 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:36.736856 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:36.737116 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:36.737119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:36.737128 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:36.737731 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.737736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.738535 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.738536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.739352 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:36.739354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:36.739931 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:36.739933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:36.740062 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:36.740725 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:36.740725 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:36.740766 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:36.740792 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:36.740961 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:36.740965 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:36.740968 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:36.741022 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:10:36.741115 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260812 00:10:36.741909 27082 heartbeater.cc:347] Connected to a master server at 127.16.99.254:43157
I20260812 00:10:36.742019 27082 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:36.742249 27082 heartbeater.cc:511] Master 127.16.99.254:43157 requested a full tablet report, sending...
I20260812 00:10:36.742743 26592 ts_manager.cc:194] Registered new tserver with Master: abef6036b02c48fca9807fdfad6f1199 (127.16.99.195:40279)
I20260812 00:10:36.743304 26592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.195:52959
I20260812 00:10:36.744928 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 26943
I20260812 00:10:36.745000 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:36.745290 16783 external_mini_cluster.cc:1428] Setting key 446689a099194158da859362e0d2d8bf
I20260812 00:10:36.746279 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:10:36.748199 26642 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:36.747688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.16.99.254:43157)
0812 00:10:36.747734 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:36.747736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:36.747883 (+ 147us) server_negotiation.cc:407] Connection header received
0812 00:10:36.747913 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:36.747914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:36.747924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:36.747977 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:36.748131 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40212: BlockingRecv error: recv got EOF from 127.0.0.1:40212 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:10:36.748329 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 26664
I20260812 00:10:36.754814 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 26805
I20260812 00:10:36.760960 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 26943
I20260812 00:10:36.768251 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 26567
2026-08-12T00:10:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (659 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T00:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:36Z Disabled control of system clock
I20260812 00:10:36.799858 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38979
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:38979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 00:10:36.885877 27093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:36.886157 27093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:36.886188 27093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:36.887468 27093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:36.887514 27093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:36.887527 27093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:36.887542 27093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:36.887553 27093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:36.888926 27093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:38979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:36.889175 27093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:36.889446 27093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:36.894580 27101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.894625 27099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:36.894567 27098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:36.895160 27093 server_base.cc:1034] running on GCE node
I20260812 00:10:36.895489 27093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:36.895756 27093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:36.896910 27093 hybrid_clock.cc:648] HybridClock initialized: now 1786493436896894 us; error 31 us; skew 500 ppm
I20260812 00:10:36.898397 27093 webserver.cc:460] Webserver started at http://127.16.99.254:39949/ using document root <none> and password file <none>
I20260812 00:10:36.898675 27093 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:36.898725 27093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:36.898890 27093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:36.899998 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "e677e2b3e4354aab9fe5157a7acddddd"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "139e4a3ffd68cebb9b195bc05e5bdf41"
server_key_iv: "887aaa146de28ebcd13cb995f30a8247"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.900363 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "e677e2b3e4354aab9fe5157a7acddddd"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "139e4a3ffd68cebb9b195bc05e5bdf41"
server_key_iv: "887aaa146de28ebcd13cb995f30a8247"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.901716 27093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:36.902678 27107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:36.902841 27093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:36.902941 27093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "e677e2b3e4354aab9fe5157a7acddddd"
format_stamp: "Formatted at 2026-08-12 00:10:36 on dist-test-slave-zpfg"
server_key: "139e4a3ffd68cebb9b195bc05e5bdf41"
server_key_iv: "887aaa146de28ebcd13cb995f30a8247"
server_key_version: "encryptionkey@0"
I20260812 00:10:36.903019 27093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:36.906754 27093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:36.907038 27093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:36.907181 27093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:36.911592 27093 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:38979
I20260812 00:10:36.911630 27163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:38979 every 8 connection(s)
I20260812 00:10:36.912001 27093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:36.912732 27164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:36.914887 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 27093
I20260812 00:10:36.914992 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:36.915172 16783 external_mini_cluster.cc:1428] Setting key 39b46015d742e491b13371ea7471f56b
I20260812 00:10:36.916211 27164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd: Bootstrap starting.
I20260812 00:10:36.916862 27164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:36.917207 27164 log.cc:824] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:36.918095 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:36.915821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.16.99.254:38979)
0812 00:10:36.916036 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:36.916044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:36.916088 (+ 44us) server_negotiation.cc:407] Connection header received
0812 00:10:36.916679 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:36.916687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:36.916731 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:36.916861 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:36.917600 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:36.917604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:36.917604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:36.917608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:36.917648 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:36.917658 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:36.917719 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:36.917842 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 00:10:36.917918 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260812 00:10:36.918241 27164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd: No bootstrap required, opened a new log
I20260812 00:10:36.919785 27164 raft_consensus.cc:348] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677e2b3e4354aab9fe5157a7acddddd" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38979 } }
I20260812 00:10:36.919973 27164 raft_consensus.cc:374] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:36.920039 27164 raft_consensus.cc:729] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e677e2b3e4354aab9fe5157a7acddddd, State: Initialized, Role: FOLLOWER
I20260812 00:10:36.920178 27164 consensus_queue.cc:260] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677e2b3e4354aab9fe5157a7acddddd" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38979 } }
I20260812 00:10:36.920252 27164 raft_consensus.cc:388] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:36.920293 27164 raft_consensus.cc:482] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:36.920349 27164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:36.921248 27164 raft_consensus.cc:504] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677e2b3e4354aab9fe5157a7acddddd" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38979 } }
I20260812 00:10:36.921350 27164 leader_election.cc:302] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e677e2b3e4354aab9fe5157a7acddddd; no voters:
I20260812 00:10:36.921645 27164 leader_election.cc:288] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:36.921706 27171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:36.921948 27164 sys_catalog.cc:564] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:36.921970 27171 raft_consensus.cc:686] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [term 1 LEADER]: Becoming Leader. State: Replica: e677e2b3e4354aab9fe5157a7acddddd, State: Running, Role: LEADER
I20260812 00:10:36.922150 27171 consensus_queue.cc:237] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677e2b3e4354aab9fe5157a7acddddd" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38979 } }
I20260812 00:10:36.922602 27173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e677e2b3e4354aab9fe5157a7acddddd. Latest consensus state: current_term: 1 leader_uuid: "e677e2b3e4354aab9fe5157a7acddddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677e2b3e4354aab9fe5157a7acddddd" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38979 } } }
I20260812 00:10:36.922760 27173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:36.922618 27172 sys_catalog.cc:455] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e677e2b3e4354aab9fe5157a7acddddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e677e2b3e4354aab9fe5157a7acddddd" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38979 } } }
I20260812 00:10:36.922838 27172 sys_catalog.cc:458] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:36.923004 27180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:36.923525 27180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:36.925225 27180 catalog_manager.cc:1347] Generated new cluster ID: ae78ebdf153f46f6a6995c341df57093
I20260812 00:10:36.925271 27180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:10:36.929054 27180 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:36.929235 27180 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:36.935050 27180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e677e2b3e4354aab9fe5157a7acddddd: Generated new TSK 0
I20260812 00:10:36.935294 27180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:10:36.943099 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38979
--builtin_ntp_servers=127.16.99.212:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 00:10:37.046309 27191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:37.046638 27191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:37.046684 27191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:37.047974 27191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:37.048043 27191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:37.048097 27191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:37.049566 27191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:37.049882 27191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:37.050218 27191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:37.055510 27196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:37.055543 27199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:37.055539 27197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:37.055871 27191 server_base.cc:1034] running on GCE node
I20260812 00:10:37.056092 27191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:37.056367 27191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:37.057530 27191 hybrid_clock.cc:648] HybridClock initialized: now 1786493437057516 us; error 32 us; skew 500 ppm
I20260812 00:10:37.058923 27191 webserver.cc:460] Webserver started at http://127.16.99.193:34023/ using document root <none> and password file <none>
I20260812 00:10:37.059173 27191 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:37.059244 27191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:37.059461 27191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:37.060674 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "445949221cd24586a1fcb8d301f2654f"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "b009f3ea718d24f6d42ae5036cd99dcf"
server_key_iv: "9604c659bae991528e0a8421e38931c8"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.061080 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "445949221cd24586a1fcb8d301f2654f"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "b009f3ea718d24f6d42ae5036cd99dcf"
server_key_iv: "9604c659bae991528e0a8421e38931c8"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.062520 27191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260812 00:10:37.063345 27205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.063531 27191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:37.063618 27191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "445949221cd24586a1fcb8d301f2654f"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "b009f3ea718d24f6d42ae5036cd99dcf"
server_key_iv: "9604c659bae991528e0a8421e38931c8"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.063689 27191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:37.068425 27191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:37.068740 27191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:37.068876 27191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:37.069084 27191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:37.069435 27191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:37.069480 27191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.069512 27191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:37.069540 27191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.077437 27191 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:44701
I20260812 00:10:37.077478 27323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:44701 every 8 connection(s)
I20260812 00:10:37.077937 27191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:37.080232 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 27191
I20260812 00:10:37.080354 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:37.080551 16783 external_mini_cluster.cc:1428] Setting key 9a23d9c05ba70edcfe00cf2946f3b7e5
I20260812 00:10:37.081667 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:37.078874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:38589 (local address 127.16.99.254:38979)
0812 00:10:37.078936 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:37.078938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:37.079166 (+ 228us) server_negotiation.cc:407] Connection header received
0812 00:10:37.079730 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:37.079732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:37.079740 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:37.079787 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:37.081327 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:37.081329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:37.081329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:37.081334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:37.081350 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:37.081355 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:37.081404 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:37.081534 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 00:10:37.081563 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:10:37.081741 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38979
--builtin_ntp_servers=127.16.99.212:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 00:10:37.082508 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:37.078845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:38979 (local address 127.16.99.193:38589)
0812 00:10:37.079042 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:37.079077 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:37.079607 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:37.079883 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:37.079887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:37.079895 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:37.081170 (+ 1275us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:37.081170 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:37.081213 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:37.081254 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:37.081432 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:37.081439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:37.081443 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:37.082116 (+ 673us) client_negotiation.cc:240] Negotiation successful
0812 00:10:37.082191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260812 00:10:37.083259 27324 heartbeater.cc:347] Connected to a master server at 127.16.99.254:38979
I20260812 00:10:37.083423 27324 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:37.083657 27324 heartbeater.cc:511] Master 127.16.99.254:38979 requested a full tablet report, sending...
I20260812 00:10:37.084213 27125 ts_manager.cc:194] Registered new tserver with Master: 445949221cd24586a1fcb8d301f2654f (127.16.99.193:44701)
W20260812 00:10:37.169226 27327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:37.169468 27327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:37.169493 27327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:37.170830 27327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:37.170881 27327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:37.170908 27327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:37.172497 27327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:37.172772 27327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:37.173095 27327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:10:37.178601 27327 server_base.cc:1034] running on GCE node
W20260812 00:10:37.178543 27333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:37.178543 27332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:37.178560 27335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:37.179051 27327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:37.179313 27327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:37.180493 27327 hybrid_clock.cc:648] HybridClock initialized: now 1786493437180479 us; error 36 us; skew 500 ppm
I20260812 00:10:37.181941 27327 webserver.cc:460] Webserver started at http://127.16.99.194:45475/ using document root <none> and password file <none>
I20260812 00:10:37.182201 27327 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:37.182250 27327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:37.182495 27327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:37.183643 27327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "4a24b43db4c043e99a4ed2abc2afd92a"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "c9a8d438933e2ecc2f1e875a83410830"
server_key_iv: "2dd7a00ff891bf4efc3723841c24fbdd"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.184031 27327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "4a24b43db4c043e99a4ed2abc2afd92a"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "c9a8d438933e2ecc2f1e875a83410830"
server_key_iv: "2dd7a00ff891bf4efc3723841c24fbdd"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.185446 27327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:10:37.186262 27341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.186472 27327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:37.186579 27327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "4a24b43db4c043e99a4ed2abc2afd92a"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "c9a8d438933e2ecc2f1e875a83410830"
server_key_iv: "2dd7a00ff891bf4efc3723841c24fbdd"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.186668 27327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:37.190261 27327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:37.190596 27327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:37.190774 27327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:37.191005 27327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:37.191349 27327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:37.191402 27327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.191444 27327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:37.191471 27327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.197753 27327 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:44881
I20260812 00:10:37.197805 27462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:44881 every 8 connection(s)
I20260812 00:10:37.198197 27327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:10:37.201455 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:37.198985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:60011 (local address 127.16.99.254:38979)
0812 00:10:37.199038 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:37.199040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:37.199392 (+ 352us) server_negotiation.cc:407] Connection header received
0812 00:10:37.199893 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:37.199895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:37.199901 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:37.199970 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:37.201250 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:37.201251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:37.201251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:37.201255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:37.201269 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:37.201273 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:37.201328 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:37.201382 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 00:10:37.201395 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:10:37.201670 27465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:37.198950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:38979 (local address 127.16.99.194:60011)
0812 00:10:37.199214 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:37.199271 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:37.199798 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:37.200053 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:37.200057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:37.200065 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:37.201093 (+ 1028us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:37.201094 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:37.201124 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:37.201157 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:37.201322 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:37.201325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:37.201327 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:10:37.201381 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:10:37.201431 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260812 00:10:37.202189 27463 heartbeater.cc:347] Connected to a master server at 127.16.99.254:38979
I20260812 00:10:37.202296 27463 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:37.202463 27463 heartbeater.cc:511] Master 127.16.99.254:38979 requested a full tablet report, sending...
I20260812 00:10:37.202844 27125 ts_manager.cc:194] Registered new tserver with Master: 4a24b43db4c043e99a4ed2abc2afd92a (127.16.99.194:44881)
I20260812 00:10:37.208207 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 27327
I20260812 00:10:37.208338 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:37.208541 16783 external_mini_cluster.cc:1428] Setting key e382fe12b91404e60534ad70a96b221a
I20260812 00:10:37.209590 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38979
--builtin_ntp_servers=127.16.99.212:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 00:10:37.297363 27466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:37.297611 27466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:37.297640 27466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:37.298929 27466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:37.298975 27466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:37.299003 27466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:37.300392 27466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:37.300607 27466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:37.300875 27466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:37.306314 27475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:37.306447 27472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:37.306447 27473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:37.306742 27466 server_base.cc:1034] running on GCE node
I20260812 00:10:37.306975 27466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:37.307189 27466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:37.308337 27466 hybrid_clock.cc:648] HybridClock initialized: now 1786493437308315 us; error 32 us; skew 500 ppm
I20260812 00:10:37.309799 27466 webserver.cc:460] Webserver started at http://127.16.99.195:33339/ using document root <none> and password file <none>
I20260812 00:10:37.310061 27466 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:37.310114 27466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:37.310213 27466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:37.311424 27466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "57ebda7a7d0947339417e1cc55137baf"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "23dd42a1bfc25ff10ece69fa1fdb8337"
server_key_iv: "e49a18b3d5d893fbd4a47536a3ce8425"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.311805 27466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "57ebda7a7d0947339417e1cc55137baf"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "23dd42a1bfc25ff10ece69fa1fdb8337"
server_key_iv: "e49a18b3d5d893fbd4a47536a3ce8425"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.313210 27466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:10:37.314172 27481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.314414 27466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:37.314520 27466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "57ebda7a7d0947339417e1cc55137baf"
format_stamp: "Formatted at 2026-08-12 00:10:37 on dist-test-slave-zpfg"
server_key: "23dd42a1bfc25ff10ece69fa1fdb8337"
server_key_iv: "e49a18b3d5d893fbd4a47536a3ce8425"
server_key_version: "encryptionkey@0"
I20260812 00:10:37.314654 27466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:37.318181 27466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:37.318517 27466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:37.318684 27466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:37.318918 27466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:37.319264 27466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:37.319319 27466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.319362 27466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:37.319388 27466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:37.325968 27466 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:44605
I20260812 00:10:37.326011 27601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:44605 every 8 connection(s)
I20260812 00:10:37.326426 27466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:10:37.329790 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:37.327261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:53959 (local address 127.16.99.254:38979)
0812 00:10:37.327348 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:37.327349 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:37.327635 (+ 286us) server_negotiation.cc:407] Connection header received
0812 00:10:37.328175 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:37.328177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:37.328185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:37.328234 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:37.329506 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:37.329508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:37.329508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:10:37.329513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:37.329529 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:10:37.329534 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:37.329608 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:37.329699 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 00:10:37.329716 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 00:10:37.330031 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:37.327208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:38979 (local address 127.16.99.195:53959)
0812 00:10:37.327510 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:37.327547 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:37.328060 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:37.328355 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:37.328377 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:37.328385 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:37.329349 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:10:37.329349 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:37.329379 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:10:37.329409 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:37.329608 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:37.329616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:37.329619 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:10:37.329710 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 00:10:37.329775 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":160,"threads_started":1}
I20260812 00:10:37.330559 27602 heartbeater.cc:347] Connected to a master server at 127.16.99.254:38979
I20260812 00:10:37.330655 27602 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:37.330789 27602 heartbeater.cc:511] Master 127.16.99.254:38979 requested a full tablet report, sending...
I20260812 00:10:37.331166 27125 ts_manager.cc:194] Registered new tserver with Master: 57ebda7a7d0947339417e1cc55137baf (127.16.99.195:44605)
I20260812 00:10:37.335934 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 27466
I20260812 00:10:37.336021 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:37.336236 16783 external_mini_cluster.cc:1428] Setting key 09f7688b95e875db24e443d035f1a91d
I20260812 00:10:37.337244 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 00:10:37.339524 27168 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:10:37.338849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.16.99.254:38979)
0812 00:10:37.338934 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:37.338936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:37.339038 (+ 102us) server_negotiation.cc:407] Connection header received
0812 00:10:37.339253 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:37.339254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:37.339263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:37.339320 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:37.339460 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56704: BlockingRecv error: recv got EOF from 127.0.0.1:56704 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 00:10:37.339699 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 27191
I20260812 00:10:37.346845 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 27327
I20260812 00:10:37.353356 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 27466
I20260812 00:10:37.358886 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 27093
2026-08-12T00:10:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (590 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:37 dist-test-slave-zpfg krb5kdc[27612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:37 dist-test-slave-zpfg krb5kdc[27612](info): set up 2 sockets
Aug 12 00:10:37 dist-test-slave-zpfg krb5kdc[27612](info): commencing operation
krb5kdc: starting...
W20260812 00:10:38.496304 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:38 dist-test-slave-zpfg krb5kdc[27612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:10:38.630069 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39401
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:39401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:38.719295 27625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:38.719553 27625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:38.719594 27625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:38.720841 27625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:10:38.720911 27625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:38.720937 27625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:38.720964 27625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:10:38.720988 27625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:10:38.722604 27625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:39401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:38.722877 27625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:38.723105 27625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:38.728055 27631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:38.728071 27632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:38.728243 27625 server_base.cc:1034] running on GCE node
W20260812 00:10:38.728071 27634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:38.728611 27625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:38.729003 27625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:38.730167 27625 hybrid_clock.cc:648] HybridClock initialized: now 1786493438730133 us; error 53 us; skew 500 ppm
Aug 12 00:10:38 dist-test-slave-zpfg krb5kdc[27612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:38.732947 27625 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:10:38.733503 27625 webserver.cc:460] Webserver started at http://127.16.99.254:33675/ using document root <none> and password file <none>
I20260812 00:10:38.733740 27625 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:38.733810 27625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:38.733963 27625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:38.735157 27625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "42ae43fd8dea4744b8d70feaaa54abe1"
format_stamp: "Formatted at 2026-08-12 00:10:38 on dist-test-slave-zpfg"
server_key: "62e6ab2fe2e927b401793101c9058ae9"
server_key_iv: "7ce20f01fcd659b172c1f19f31f2e9fb"
server_key_version: "encryptionkey@0"
I20260812 00:10:38.735543 27625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "42ae43fd8dea4744b8d70feaaa54abe1"
format_stamp: "Formatted at 2026-08-12 00:10:38 on dist-test-slave-zpfg"
server_key: "62e6ab2fe2e927b401793101c9058ae9"
server_key_iv: "7ce20f01fcd659b172c1f19f31f2e9fb"
server_key_version: "encryptionkey@0"
I20260812 00:10:38.736936 27625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:10:38.737727 27641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:38.737905 27625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:38.738009 27625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "42ae43fd8dea4744b8d70feaaa54abe1"
format_stamp: "Formatted at 2026-08-12 00:10:38 on dist-test-slave-zpfg"
server_key: "62e6ab2fe2e927b401793101c9058ae9"
server_key_iv: "7ce20f01fcd659b172c1f19f31f2e9fb"
server_key_version: "encryptionkey@0"
I20260812 00:10:38.738081 27625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:38.771198 27625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:38.774693 27625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:38.774981 27625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:38.780848 27625 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:39401
I20260812 00:10:38.780884 27694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:39401 every 8 connection(s)
I20260812 00:10:38.781296 27625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:10:38.782439 27695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:38.784968 27695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1: Bootstrap starting.
I20260812 00:10:38.785593 27695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:38.785843 27695 log.cc:824] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:38.786746 27695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1: No bootstrap required, opened a new log
I20260812 00:10:38.786724 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 27625
I20260812 00:10:38.786878 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:10:38.787082 16783 external_mini_cluster.cc:1428] Setting key 48cc8105c8c30d9e2b531b2be32fa0c3
I20260812 00:10:38.788820 27695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39401 } }
I20260812 00:10:38.788975 27695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:38.788998 27695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ae43fd8dea4744b8d70feaaa54abe1, State: Initialized, Role: FOLLOWER
I20260812 00:10:38.789160 27695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39401 } }
I20260812 00:10:38.789297 27695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:10:38.789386 27695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:10:38.789451 27695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:38.790658 27695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39401 } }
I20260812 00:10:38.790758 27695 leader_election.cc:302] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42ae43fd8dea4744b8d70feaaa54abe1; no voters:
I20260812 00:10:38.790997 27695 leader_election.cc:288] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:10:38.791020 27700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:38.791144 27700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [term 1 LEADER]: Becoming Leader. State: Replica: 42ae43fd8dea4744b8d70feaaa54abe1, State: Running, Role: LEADER
I20260812 00:10:38.791230 27700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39401 } }
I20260812 00:10:38.791286 27695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:10:38.791591 27700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42ae43fd8dea4744b8d70feaaa54abe1. Latest consensus state: current_term: 1 leader_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39401 } } }
I20260812 00:10:38.791666 27700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:38.791953 27704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:10:38.792698 27702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ae43fd8dea4744b8d70feaaa54abe1" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 39401 } } }
I20260812 00:10:38.792784 27702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:10:38.792892 27704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:10:38.794564 27704 catalog_manager.cc:1347] Generated new cluster ID: 74bc5fb89a074572a2bf5da1d8cf0ea0
I20260812 00:10:38.794625 27704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:10:38 dist-test-slave-zpfg krb5kdc[27612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:38.800787 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:38.787683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.16.99.254:39401)
0812 00:10:38.787850 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:38.787857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:38.787883 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:10:38.788826 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:38.788836 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:38.788899 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:38.789211 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:38.790051 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:38.791392 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:38.792461 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:38.793107 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:38.793215 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:38.796407 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:38.796412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:38.796412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:38.796428 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:38.798679 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:38.799116 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:38.799120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:38.799120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:38.799296 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:38.799508 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:38.799509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:38.799509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:38.799894 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:38.800098 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:38.800185 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 00:10:38.800358 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260812 00:10:38.803364 27704 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:10:38.803984 27704 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:10:38.807678 27704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1: Generated new TSK 0
I20260812 00:10:38.807871 27704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:10:38.837561 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39401
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:38.926473 27722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:38.926728 27722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:38.926759 27722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:38.928089 27722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:38.928141 27722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:38.928172 27722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:10:38.929865 27722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:38.930107 27722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:38.930322 27722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:38.935822 27730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:38.935873 27728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:38.935823 27727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:38.936133 27722 server_base.cc:1034] running on GCE node
I20260812 00:10:38.936357 27722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:38.936589 27722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:38.937868 27722 hybrid_clock.cc:648] HybridClock initialized: now 1786493438937846 us; error 36 us; skew 500 ppm
Aug 12 00:10:38 dist-test-slave-zpfg krb5kdc[27612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:38.946425 27722 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:10:38.946945 27722 webserver.cc:460] Webserver started at http://127.16.99.193:44875/ using document root <none> and password file <none>
I20260812 00:10:38.947168 27722 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:38.947216 27722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:38.947330 27722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:38.948503 27722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "0be1c99af3e146fcb5fd850d837d0046"
format_stamp: "Formatted at 2026-08-12 00:10:38 on dist-test-slave-zpfg"
server_key: "c1d5a2994bea297575a7ac3ae3c8e4b0"
server_key_iv: "bc24c2f5eb1ddc7c06aa2fc0fee8fd8d"
server_key_version: "encryptionkey@0"
I20260812 00:10:38.948865 27722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "0be1c99af3e146fcb5fd850d837d0046"
format_stamp: "Formatted at 2026-08-12 00:10:38 on dist-test-slave-zpfg"
server_key: "c1d5a2994bea297575a7ac3ae3c8e4b0"
server_key_iv: "bc24c2f5eb1ddc7c06aa2fc0fee8fd8d"
server_key_version: "encryptionkey@0"
I20260812 00:10:38.950510 27722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:10:38.951332 27738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:38.951498 27722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:38.951604 27722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "0be1c99af3e146fcb5fd850d837d0046"
format_stamp: "Formatted at 2026-08-12 00:10:38 on dist-test-slave-zpfg"
server_key: "c1d5a2994bea297575a7ac3ae3c8e4b0"
server_key_iv: "bc24c2f5eb1ddc7c06aa2fc0fee8fd8d"
server_key_version: "encryptionkey@0"
I20260812 00:10:38.951682 27722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:38.969305 27722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:38.971676 27722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:38.971851 27722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:38.972112 27722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:38.972440 27722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:38.972493 27722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:38.972534 27722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:38.972566 27722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:38.978691 27722 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:44729
I20260812 00:10:38.978780 27859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:44729 every 8 connection(s)
I20260812 00:10:38.979103 27722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:10:38.984023 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 27722
I20260812 00:10:38.984136 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:10:38.984398 16783 external_mini_cluster.cc:1428] Setting key ebff88b361c0035f5f8d8610c9e2ce9a
Aug 12 00:10:38 dist-test-slave-zpfg krb5kdc[27612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493438, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:38.990067 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:38.979899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:41345 (local address 127.16.99.254:39401)
0812 00:10:38.979965 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:38.979967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:38.980141 (+ 174us) server_negotiation.cc:407] Connection header received
0812 00:10:38.980698 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:38.980699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:38.980714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:38.980783 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:38.982081 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:38.982728 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:38.983773 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:38.984130 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:38.984213 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:38.986620 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:38.986622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:38.986622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:38.986630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:38.988315 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:38.988904 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:38.988905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:38.988905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:38.988971 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:38.989277 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:38.989278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:38.989278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:38.989461 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:38.989519 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:38.989910 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 00:10:38.989996 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:10:38.990398 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:38.979818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39401 (local address 127.16.99.193:41345)
0812 00:10:38.980006 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:38.980050 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:38.980564 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:38.980928 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:38.980931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:38.981760 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:38.981968 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:38.981974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:38.982817 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:38.982819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:38.983675 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:38.983677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:38.984254 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:38.984255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:38.984376 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:38.985002 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:38.985029 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:38.986518 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:38.988404 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:38.988414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:38.988416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:38.988758 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:38.989043 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:38.989044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:38.989045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:38.989206 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:38.989533 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:38.989542 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:38.989739 (+ 197us) client_negotiation.cc:769] Sending connection context
0812 00:10:38.989882 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 00:10:38.990083 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260812 00:10:38.990968 27860 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39401
I20260812 00:10:38.991091 27860 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:38.991293 27860 heartbeater.cc:511] Master 127.16.99.254:39401 requested a full tablet report, sending...
I20260812 00:10:38.991928 27658 ts_manager.cc:194] Registered new tserver with Master: 0be1c99af3e146fcb5fd850d837d0046 (127.16.99.193:44729)
I20260812 00:10:38.992640 27658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:41345
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:10:39.017968 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39401
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:39.103718 27865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:39.103986 27865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:39.104017 27865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:39.105305 27865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:39.105355 27865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:39.105386 27865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:10:39.107000 27865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:39.107280 27865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:39.107534 27865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:39.113505 27870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:39.113660 27865 server_base.cc:1034] running on GCE node
W20260812 00:10:39.113508 27874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:39.113543 27871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:39.114030 27865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:39.114250 27865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:39.115437 27865 hybrid_clock.cc:648] HybridClock initialized: now 1786493439115413 us; error 32 us; skew 500 ppm
Aug 12 00:10:39 dist-test-slave-zpfg krb5kdc[27612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493439, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:39.118670 27865 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:10:39.119226 27865 webserver.cc:460] Webserver started at http://127.16.99.194:35521/ using document root <none> and password file <none>
I20260812 00:10:39.119442 27865 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:39.119488 27865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:39.119594 27865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:39.120790 27865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "121e688fa21a42e8b3d2342e84ee7c10"
format_stamp: "Formatted at 2026-08-12 00:10:39 on dist-test-slave-zpfg"
server_key: "2462f3160719d79d322cae8db2dd09f1"
server_key_iv: "7c2502afe0ce7e55225491b58c4c14b7"
server_key_version: "encryptionkey@0"
I20260812 00:10:39.121152 27865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "121e688fa21a42e8b3d2342e84ee7c10"
format_stamp: "Formatted at 2026-08-12 00:10:39 on dist-test-slave-zpfg"
server_key: "2462f3160719d79d322cae8db2dd09f1"
server_key_iv: "7c2502afe0ce7e55225491b58c4c14b7"
server_key_version: "encryptionkey@0"
I20260812 00:10:39.122789 27865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:10:39.123744 27882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:39.123971 27865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:39.124190 27865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "121e688fa21a42e8b3d2342e84ee7c10"
format_stamp: "Formatted at 2026-08-12 00:10:39 on dist-test-slave-zpfg"
server_key: "2462f3160719d79d322cae8db2dd09f1"
server_key_iv: "7c2502afe0ce7e55225491b58c4c14b7"
server_key_version: "encryptionkey@0"
I20260812 00:10:39.124274 27865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:39.134948 27865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:39.137203 27865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:39.137413 27865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:39.137687 27865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:39.138028 27865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:39.138083 27865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:39.138125 27865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:39.138151 27865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:39.144629 27865 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:38373
I20260812 00:10:39.144672 28002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:38373 every 8 connection(s)
I20260812 00:10:39.145009 27865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:10:39 dist-test-slave-zpfg krb5kdc[27612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493439, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:39.154680 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 27865
I20260812 00:10:39.154796 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:10:39.154983 16783 external_mini_cluster.cc:1428] Setting key 0e48d93c2d33fdb7180684a798f723db
I20260812 00:10:39.155256 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.145763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:38613 (local address 127.16.99.254:39401)
0812 00:10:39.145829 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.145831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.146215 (+ 384us) server_negotiation.cc:407] Connection header received
0812 00:10:39.146863 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.146865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.146876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.146941 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:39.147923 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.148455 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.149444 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.149633 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.149691 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.152313 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:39.152315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:39.152315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:39.152323 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:39.153692 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:39.154070 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:39.154071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:39.154071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:39.154132 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:39.154499 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:39.154501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:39.154501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:39.154662 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:39.154737 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.155067 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.155164 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:10:39.155468 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.145758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39401 (local address 127.16.99.194:38613)
0812 00:10:39.146052 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.146103 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:39.146712 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:39.147050 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:39.147054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:39.147640 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:39.147838 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.147843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.148540 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.148541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.149354 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.149356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.149749 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.149750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.149868 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.150590 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:39.150599 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:39.152157 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:39.153767 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:39.153773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:39.153776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:39.154006 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:39.154185 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:39.154186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:39.154186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:39.154414 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:39.154738 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:39.154746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:39.154927 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 00:10:39.155032 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 00:10:39.155203 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260812 00:10:39.155956 28003 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39401
I20260812 00:10:39.156075 28003 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:39.156255 28003 heartbeater.cc:511] Master 127.16.99.254:39401 requested a full tablet report, sending...
I20260812 00:10:39.156656 27658 ts_manager.cc:194] Registered new tserver with Master: 121e688fa21a42e8b3d2342e84ee7c10 (127.16.99.194:38373)
I20260812 00:10:39.157192 27658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:38613
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:10:39.185786 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39401
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:10:39.275283 28008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:10:39.275516 28008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:10:39.275542 28008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:10:39.276877 28008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:10:39.276938 28008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:10:39.276968 28008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:10:39.278645 28008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:10:39.278915 28008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:10:39.279170 28008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:10:39.284713 28013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:39.284703 28014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:10:39.284708 28016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:10:39.285085 28008 server_base.cc:1034] running on GCE node
I20260812 00:10:39.285305 28008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:10:39.285548 28008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:10:39.286693 28008 hybrid_clock.cc:648] HybridClock initialized: now 1786493439286673 us; error 39 us; skew 500 ppm
Aug 12 00:10:39 dist-test-slave-zpfg krb5kdc[27612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493439, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:10:39.289455 28008 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:10:39.289961 28008 webserver.cc:460] Webserver started at http://127.16.99.195:43621/ using document root <none> and password file <none>
I20260812 00:10:39.290196 28008 fs_manager.cc:359] Metadata directory not provided
I20260812 00:10:39.290264 28008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:10:39.290436 28008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:10:39.291622 28008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "7b6177571e3b4bb490f30422b40fbc71"
format_stamp: "Formatted at 2026-08-12 00:10:39 on dist-test-slave-zpfg"
server_key: "a30003caa9d1be5d4548ac50652b2597"
server_key_iv: "2fb0f7183df93adb14085116f5d0f54d"
server_key_version: "encryptionkey@0"
I20260812 00:10:39.291992 28008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "7b6177571e3b4bb490f30422b40fbc71"
format_stamp: "Formatted at 2026-08-12 00:10:39 on dist-test-slave-zpfg"
server_key: "a30003caa9d1be5d4548ac50652b2597"
server_key_iv: "2fb0f7183df93adb14085116f5d0f54d"
server_key_version: "encryptionkey@0"
I20260812 00:10:39.293619 28008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:10:39.294531 28023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:39.294759 28008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:10:39.294893 28008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "7b6177571e3b4bb490f30422b40fbc71"
format_stamp: "Formatted at 2026-08-12 00:10:39 on dist-test-slave-zpfg"
server_key: "a30003caa9d1be5d4548ac50652b2597"
server_key_iv: "2fb0f7183df93adb14085116f5d0f54d"
server_key_version: "encryptionkey@0"
I20260812 00:10:39.294984 28008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:10:39.303586 28008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:10:39.305877 28008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:10:39.306052 28008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:10:39.306438 28008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:10:39.306772 28008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:10:39.306826 28008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:39.306869 28008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:10:39.306895 28008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:10:39.313328 28008 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:41739
I20260812 00:10:39.313402 28146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:41739 every 8 connection(s)
I20260812 00:10:39.313731 28008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:10:39 dist-test-slave-zpfg krb5kdc[27612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493439, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:10:39.322441 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 28008
I20260812 00:10:39.322561 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:10:39.322739 16783 external_mini_cluster.cc:1428] Setting key 892a29e083fb94776f62867a4f010fbd
I20260812 00:10:39.325037 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.314529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:56285 (local address 127.16.99.254:39401)
0812 00:10:39.314586 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.314588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.314857 (+ 269us) server_negotiation.cc:407] Connection header received
0812 00:10:39.315432 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.315434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.315446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.315500 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:39.316551 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.317075 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.318515 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.318818 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.318913 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.321721 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:39.321723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:39.321723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:39.321738 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:39.323516 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:39.324010 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:39.324011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:39.324011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:39.324077 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:39.324348 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:39.324349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:39.324349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:39.324479 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:39.324549 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.324878 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.324969 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 00:10:39.325258 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.314564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:39401 (local address 127.16.99.195:56285)
0812 00:10:39.314730 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.314770 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:39.315308 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:39.315586 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:39.315590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:39.316247 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:10:39.316448 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.316453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.317174 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.317175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.318323 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.318327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.318923 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.318925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.319080 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.319708 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:10:39.319717 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:10:39.321574 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:10:39.323590 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:39.323601 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:39.323604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:39.323920 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:39.324148 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:10:39.324148 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:10:39.324149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:10:39.324273 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:10:39.324552 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:10:39.324560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:10:39.324711 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 00:10:39.324835 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 00:10:39.325023 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260812 00:10:39.325747 28147 heartbeater.cc:347] Connected to a master server at 127.16.99.254:39401
I20260812 00:10:39.325848 28147 heartbeater.cc:464] Registering TS with master...
I20260812 00:10:39.326010 28147 heartbeater.cc:511] Master 127.16.99.254:39401 requested a full tablet report, sending...
I20260812 00:10:39.326457 27658 ts_manager.cc:194] Registered new tserver with Master: 7b6177571e3b4bb490f30422b40fbc71 (127.16.99.195:41739)
I20260812 00:10:39.326862 27658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:56285
I20260812 00:10:39.335381 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:10:39.343637 27699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.337055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.16.99.254:39401)
0812 00:10:39.337230 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.337232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.337241 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:39.337264 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.337265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.337274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.337327 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:10:39.337993 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.338615 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.339390 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.339627 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.339698 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.341001 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:10:39.341003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:10:39.341003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:10:39.341010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:10:39.342586 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:39.342897 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:39.342898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:39.342898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:39.342951 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:10:39.343101 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:10:39.343101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:10:39.343101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:10:39.343243 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:10:39.343375 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.343511 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.343572 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260812 00:10:39.345323 27658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:10:39.347200 27658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:10:39.353658 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.349664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:38373 (local address 127.0.0.1:60378)
0812 00:10:39.349850 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.349894 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:39.349964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:39.350183 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:39.350185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:39.350187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:39.350267 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.350269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.351442 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.351443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.352387 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.352389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.353349 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.353349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.353428 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.353445 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 00:10:39.353526 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:10:39.353581 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":64,"threads_started":1}
I20260812 00:10:39.353670 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.349796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.16.99.194:38373)
0812 00:10:39.350013 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.350019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.350029 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:10:39.350068 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.350070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.350085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.350179 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:39.350358 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.351343 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.352485 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.353221 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.353303 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.353367 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.353542 (+ 175us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.353584 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260812 00:10:39.354224 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.349673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:41739 (local address 127.0.0.1:48566)
0812 00:10:39.350292 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.350302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:39.350392 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:39.350718 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:39.350718 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:39.350720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:39.350791 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.350793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.352162 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.352163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.353072 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.353074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.353986 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.353987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.354049 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.354054 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:10:39.354103 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:10:39.354151 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260812 00:10:39.354259 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.349765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.16.99.195:41739)
0812 00:10:39.350028 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.350033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.350383 (+ 350us) server_negotiation.cc:407] Connection header received
0812 00:10:39.350540 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.350543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.350560 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.350636 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:39.350872 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.352053 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.353163 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.353898 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.353985 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.354094 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.354122 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.354167 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":123,"threads_started":1}
I20260812 00:10:39.354460 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.349877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:44729 (local address 127.0.0.1:51856)
0812 00:10:39.350378 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.350388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:39.350437 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:39.350743 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:39.350744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:39.350746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:39.350809 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.350811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.352124 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.352125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.353224 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.353226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.354150 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.354151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.354209 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.354214 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:10:39.354264 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 00:10:39.354386 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260812 00:10:39.354964 28075 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6b00ab6f78496c9ce691201c5b21a3 (DEFAULT_TABLE table=test-table [id=6c06e9db6f5f42ec8a09cff0e966973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:39.354975 27933 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6b00ab6f78496c9ce691201c5b21a3 (DEFAULT_TABLE table=test-table [id=6c06e9db6f5f42ec8a09cff0e966973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:39.355345 27933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6b00ab6f78496c9ce691201c5b21a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:39.355345 28075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6b00ab6f78496c9ce691201c5b21a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:39.358192 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.350101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.16.99.193:44729)
0812 00:10:39.350257 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.350261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.350500 (+ 239us) server_negotiation.cc:407] Connection header received
0812 00:10:39.350548 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.350550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.350566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.350655 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:39.350915 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.352011 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.353312 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.354000 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.354061 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.354112 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.357882 (+ 3770us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.358100 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260812 00:10:39.358168 28166 tablet_bootstrap.cc:492] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Bootstrap starting.
I20260812 00:10:39.358448 28165 tablet_bootstrap.cc:492] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Bootstrap starting.
I20260812 00:10:39.358825 27791 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6b00ab6f78496c9ce691201c5b21a3 (DEFAULT_TABLE table=test-table [id=6c06e9db6f5f42ec8a09cff0e966973c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:10:39.358961 28166 tablet_bootstrap.cc:654] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:39.359094 28165 tablet_bootstrap.cc:654] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:39.359169 27791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6b00ab6f78496c9ce691201c5b21a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:10:39.359587 28166 log.cc:824] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:39.359679 28165 log.cc:824] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:39.360402 28165 tablet_bootstrap.cc:492] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: No bootstrap required, opened a new log
I20260812 00:10:39.360410 28166 tablet_bootstrap.cc:492] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: No bootstrap required, opened a new log
I20260812 00:10:39.360476 28165 ts_tablet_manager.cc:1397] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:10:39.360476 28166 ts_tablet_manager.cc:1397] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:39.362112 28169 tablet_bootstrap.cc:492] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046: Bootstrap starting.
I20260812 00:10:39.362485 28165 raft_consensus.cc:348] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.362485 28166 raft_consensus.cc:348] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.362670 28166 raft_consensus.cc:374] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:39.362670 28165 raft_consensus.cc:374] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:39.362709 28166 raft_consensus.cc:729] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6177571e3b4bb490f30422b40fbc71, State: Initialized, Role: FOLLOWER
I20260812 00:10:39.362709 28165 raft_consensus.cc:729] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121e688fa21a42e8b3d2342e84ee7c10, State: Initialized, Role: FOLLOWER
I20260812 00:10:39.362844 28169 tablet_bootstrap.cc:654] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046: Neither blocks nor log segments found. Creating new log.
I20260812 00:10:39.362838 28165 consensus_queue.cc:260] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.362866 28166 consensus_queue.cc:260] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.363142 28165 ts_tablet_manager.cc:1428] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 00:10:39.363193 28003 heartbeater.cc:503] Master 127.16.99.254:39401 was elected leader, sending a full tablet report...
I20260812 00:10:39.363142 28166 ts_tablet_manager.cc:1428] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:10:39.363396 28147 heartbeater.cc:503] Master 127.16.99.254:39401 was elected leader, sending a full tablet report...
I20260812 00:10:39.363451 28169 log.cc:824] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046: Log is configured to *not* fsync() on all Append() calls
I20260812 00:10:39.364307 28169 tablet_bootstrap.cc:492] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046: No bootstrap required, opened a new log
I20260812 00:10:39.364388 28169 ts_tablet_manager.cc:1397] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:39.365859 28169 raft_consensus.cc:348] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.365996 28169 raft_consensus.cc:374] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:10:39.366021 28169 raft_consensus.cc:729] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be1c99af3e146fcb5fd850d837d0046, State: Initialized, Role: FOLLOWER
I20260812 00:10:39.366156 28169 consensus_queue.cc:260] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.366456 28169 ts_tablet_manager.cc:1428] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:10:39.366504 27860 heartbeater.cc:503] Master 127.16.99.254:39401 was elected leader, sending a full tablet report...
I20260812 00:10:39.393265 28170 raft_consensus.cc:482] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:10:39.393452 28170 raft_consensus.cc:504] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.393946 28170 leader_election.cc:288] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 121e688fa21a42e8b3d2342e84ee7c10 (127.16.99.194:38373), 0be1c99af3e146fcb5fd850d837d0046 (127.16.99.193:44729)
W20260812 00:10:39.395653 28004 tablet.cc:2369] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:10:39.397475 28164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.394374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40027 (local address 127.16.99.193:44729)
0812 00:10:39.394427 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.394429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.394526 (+ 97us) server_negotiation.cc:407] Connection header received
0812 00:10:39.394673 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.394674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.394685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.394745 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:39.394927 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.395615 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.396727 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.397124 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.397208 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.397218 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.397363 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.397404 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:10:39.397495 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.394192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:44729 (local address 127.16.99.195:40027)
0812 00:10:39.394446 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.394459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:39.394523 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:39.394761 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:39.394762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:39.394764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:39.394858 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.394859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.395697 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.395698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.396628 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.396631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.397207 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.397208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.397291 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.397295 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:39.397359 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 00:10:39.397402 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":45,"thread_start_us":132,"threads_started":1}
I20260812 00:10:39.397634 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.394240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:53427 (local address 127.16.99.194:38373)
0812 00:10:39.394301 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.394304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.394318 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:10:39.394440 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.394441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.394454 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.394528 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:10:39.394762 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.395707 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.396802 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.397236 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.397328 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.397337 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.397515 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.397569 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:10:39.397634 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.394156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:38373 (local address 127.16.99.195:53427)
0812 00:10:39.394247 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.394259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:10:39.394310 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:10:39.394558 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:10:39.394560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:10:39.394562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:10:39.394655 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.394658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.395820 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.395821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.396689 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:10:39.396692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.397333 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.397334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:10:39.397444 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.397448 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:10:39.397508 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:10:39.397556 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260812 00:10:39.397893 27812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6b00ab6f78496c9ce691201c5b21a3" candidate_uuid: "7b6177571e3b4bb490f30422b40fbc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be1c99af3e146fcb5fd850d837d0046" is_pre_election: true
I20260812 00:10:39.398079 27812 raft_consensus.cc:2454] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6177571e3b4bb490f30422b40fbc71 in term 0.
I20260812 00:10:39.398363 28025 leader_election.cc:302] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be1c99af3e146fcb5fd850d837d0046, 7b6177571e3b4bb490f30422b40fbc71; no voters:
I20260812 00:10:39.398490 28170 raft_consensus.cc:2781] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:10:39.398547 28170 raft_consensus.cc:482] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:10:39.398571 28170 raft_consensus.cc:3037] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:39.398597 27953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6b00ab6f78496c9ce691201c5b21a3" candidate_uuid: "7b6177571e3b4bb490f30422b40fbc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121e688fa21a42e8b3d2342e84ee7c10" is_pre_election: true
I20260812 00:10:39.398792 27953 raft_consensus.cc:2454] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6177571e3b4bb490f30422b40fbc71 in term 0.
I20260812 00:10:39.399477 28170 raft_consensus.cc:504] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.399629 28170 leader_election.cc:288] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [CANDIDATE]: Term 1 election: Requested vote from peers 121e688fa21a42e8b3d2342e84ee7c10 (127.16.99.194:38373), 0be1c99af3e146fcb5fd850d837d0046 (127.16.99.193:44729)
I20260812 00:10:39.399845 27953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6b00ab6f78496c9ce691201c5b21a3" candidate_uuid: "7b6177571e3b4bb490f30422b40fbc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121e688fa21a42e8b3d2342e84ee7c10"
I20260812 00:10:39.399932 27953 raft_consensus.cc:3037] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:39.400741 27953 raft_consensus.cc:2454] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6177571e3b4bb490f30422b40fbc71 in term 1.
I20260812 00:10:39.400897 27812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6b00ab6f78496c9ce691201c5b21a3" candidate_uuid: "7b6177571e3b4bb490f30422b40fbc71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be1c99af3e146fcb5fd850d837d0046"
I20260812 00:10:39.400940 28024 leader_election.cc:302] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 121e688fa21a42e8b3d2342e84ee7c10, 7b6177571e3b4bb490f30422b40fbc71; no voters:
I20260812 00:10:39.401022 28170 raft_consensus.cc:2781] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:10:39.400985 27812 raft_consensus.cc:3037] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:10:39.401207 28170 raft_consensus.cc:686] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 1 LEADER]: Becoming Leader. State: Replica: 7b6177571e3b4bb490f30422b40fbc71, State: Running, Role: LEADER
I20260812 00:10:39.401331 28170 consensus_queue.cc:237] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } }
I20260812 00:10:39.401983 27812 raft_consensus.cc:2454] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6177571e3b4bb490f30422b40fbc71 in term 1.
I20260812 00:10:39.402551 27658 catalog_manager.cc:5495] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b6177571e3b4bb490f30422b40fbc71 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "7b6177571e3b4bb490f30422b40fbc71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b6177571e3b4bb490f30422b40fbc71" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 41739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 } health_report { overall_health: UNKNOWN } } }
I20260812 00:10:39.412192 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.409579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.16.99.195:41739)
0812 00:10:39.409638 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:10:39.409640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:10:39.409649 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:10:39.409690 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:10:39.409690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:10:39.409713 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:10:39.409791 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:10:39.410005 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.410607 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.411393 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:10:39.411583 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:10:39.411653 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:10:39.411830 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:10:39.411933 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:10:39.412018 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:10:39.412058 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 00:10:39.412081 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:10:39.414083 27953 raft_consensus.cc:1261] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 1 FOLLOWER]: Refusing update from remote peer 7b6177571e3b4bb490f30422b40fbc71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:39.414166 27812 raft_consensus.cc:1261] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046 [term 1 FOLLOWER]: Refusing update from remote peer 7b6177571e3b4bb490f30422b40fbc71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:10:39.414543 28175 consensus_queue.cc:1035] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121e688fa21a42e8b3d2342e84ee7c10" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 38373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:39.414693 28170 consensus_queue.cc:1035] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be1c99af3e146fcb5fd850d837d0046" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:10:39.417250 28178 mvcc.cc:205] Tried to move back new op lower bound from 7317477127837138944 to 7317477127788666880. Current Snapshot: MvccSnapshot[applied={T|T < 7317477127837138944}]
I20260812 00:10:39.420094 27657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:39.420197 27657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:10:39.421016 27657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42ae43fd8dea4744b8d70feaaa54abe1: Sending DeleteTablet for 3 replicas of tablet 7c6b00ab6f78496c9ce691201c5b21a3
I20260812 00:10:39.421401 28075 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6b00ab6f78496c9ce691201c5b21a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:39 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:48566
I20260812 00:10:39.421461 27791 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6b00ab6f78496c9ce691201c5b21a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:39 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51856
I20260812 00:10:39.421600 27933 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6b00ab6f78496c9ce691201c5b21a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:10:39 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:60378
I20260812 00:10:39.421713 28189 tablet_replica.cc:330] T 7c6b00ab6f78496c9ce691201c5b21a3 P 0be1c99af3e146fcb5fd850d837d0046: stopping tablet replica
I20260812 00:10:39.421905 28190 tablet_replica.cc:330] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: stopping tablet replica
I20260812 00:10:39.422008 28190 raft_consensus.cc:2229] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:10:39.422111 28190 raft_consensus.cc:2258] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:39.422156 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 27722
I20260812 00:10:39.423434 28190 ts_tablet_manager.cc:1905] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:10:39.425556 28190 ts_tablet_manager.cc:1918] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:39.425673 28190 log.cc:1196] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/7c6b00ab6f78496c9ce691201c5b21a3
I20260812 00:10:39.425990 28190 ts_tablet_manager.cc:1939] T 7c6b00ab6f78496c9ce691201c5b21a3 P 121e688fa21a42e8b3d2342e84ee7c10: Deleting consensus metadata
I20260812 00:10:39.426662 27642 catalog_manager.cc:4841] TS 121e688fa21a42e8b3d2342e84ee7c10 (127.16.99.194:38373): tablet 7c6b00ab6f78496c9ce691201c5b21a3 (table test-table [id=6c06e9db6f5f42ec8a09cff0e966973c]) successfully deleted
I20260812 00:10:39.430511 28188 tablet_replica.cc:330] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: stopping tablet replica
I20260812 00:10:39.430668 28188 raft_consensus.cc:2229] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:10:39.430795 28188 raft_consensus.cc:2258] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:10:39.431342 28188 ts_tablet_manager.cc:1905] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 00:10:39.432118 27643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:44729 (error 108)
I20260812 00:10:39.432298 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 27865
I20260812 00:10:39.433786 28188 ts_tablet_manager.cc:1918] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:10:39.433866 28188 log.cc:1196] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/7c6b00ab6f78496c9ce691201c5b21a3
I20260812 00:10:39.434156 28188 ts_tablet_manager.cc:1939] T 7c6b00ab6f78496c9ce691201c5b21a3 P 7b6177571e3b4bb490f30422b40fbc71: Deleting consensus metadata
I20260812 00:10:39.434662 27643 catalog_manager.cc:4841] TS 7b6177571e3b4bb490f30422b40fbc71 (127.16.99.195:41739): tablet 7c6b00ab6f78496c9ce691201c5b21a3 (table test-table [id=6c06e9db6f5f42ec8a09cff0e966973c]) successfully deleted
I20260812 00:10:39.439229 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:10:39.439035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:44729 (local address 127.0.0.1:51864)
0812 00:10:39.439121 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0812 00:10:39.439185 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:44729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 00:10:39.439409 27643 catalog_manager.cc:4584] TS 0be1c99af3e146fcb5fd850d837d0046 (127.16.99.193:44729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c6b00ab6f78496c9ce691201c5b21a3: Network error: Client connection negotiation failed: client connection to 127.16.99.193:44729: connect: Connection refused (error 111)
I20260812 00:10:39.443789 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 28008
I20260812 00:10:39.450440 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 27625
2026-08-12T00:10:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2092 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:10:39 dist-test-slave-zpfg krb5kdc[28194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:10:39 dist-test-slave-zpfg krb5kdc[28194](info): set up 2 sockets
Aug 12 00:10:39 dist-test-slave-zpfg krb5kdc[28194](info): commencing operation
krb5kdc: starting...
W20260812 00:10:40.534960 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:10:40 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:10:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/rangeradmin_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/rangeradmin_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/rangerlookup_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/rangerlookup_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
I20260812 00:10:40.721663 16783 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcVEJCj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/postgres -l logfile start
2026-08-12 00:10:43.710 UTC [28228] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 00:10:43.710 UTC [28228] LOG: listening on IPv4 address "127.16.99.212", port 48921
2026-08-12 00:10:43.715 UTC [28228] LOG: listening on Unix socket "/tmp/.s.PGSQL.48921"
2026-08-12 00:10:43.739 UTC [28231] LOG: database system was shut down at 2026-08-12 00:10:41 UTC
2026-08-12 00:10:43.744 UTC [28228] LOG: database system is ready to accept connections
I20260812 00:10:44.739881 16783 mini_postgres.cc:96] Postgres bound to 48921
2026-08-12 00:10:44.748 UTC [28240] FATAL: database "slave" does not exist
127.16.99.212:48921 - accepting connections
I20260812 00:10:44.748977 16783 mini_ranger.cc:161] Starting Ranger...
I20260812 00:10:44.766129 16783 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 00:10:45.379738 16783 mini_ranger.cc:87] Created ranger Postgres database
I20260812 00:10:45.379837 16783 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-admin
2026-08-12 00:10:45,751 [I] DB FLAVOR :POSTGRES
2026-08-12 00:10:45,751 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:10:45,751 [I] Checking connection..
2026-08-12 00:10:45,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 00:10:46,036 [I] Checking connection passed.
2026-08-12 00:10:46,036 [I] --------- Verifying version history table ---------
2026-08-12 00:10:46,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:10:46,290 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:10:46,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:10:46,576 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:10:46,576 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 00:10:46,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 00:10:46.865 UTC [28308] WARNING: there is no transaction in progress
2026-08-12 00:10:46,873 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 00:10:46,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:10:47,140 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 00:10:47,140 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:10:47,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 00:10:47,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 00:10:47,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpfg.c.gcp-upstream.internal','N') ;"
2026-08-12 00:10:47,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:10:48,198 [I] Table x_portal_user does not exist in database ranger
2026-08-12 00:10:48,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:10:48,454 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 00:10:48,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:10:48,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:10:49,003 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 00:10:49,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 00:10:50.211 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.231 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.251 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.269 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.289 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.311 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.375 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.384 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.403 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.416 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.428 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.445 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.458 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.467 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.476 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:50.485 UTC [28422] WARNING: there is no transaction in progress
2026-08-12 00:10:51,381 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 00:10:51,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 00:10:51,652 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 00:10:51,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:10:51,922 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 00:10:51,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:10:52,235 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 00:10:52,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:10:52,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:10:52,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpfg.c.gcp-upstream.internal';"
2026-08-12 00:10:53,074 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 00:10:53,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 00:10:53,360 [I] DB_PATCHES have already been applied
I20260812 00:10:53.367367 16783 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcVEJCj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 00:10:53.367445 16783 mini_ranger.cc:193] Using host: 127.16.99.212
I20260812 00:10:53.371665 16783 mini_ranger.cc:227] Ranger admin URL: http://127.16.99.212:46439
Aug 12, 2026 12:10:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:10:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:10:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 12:10:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 12:10:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 12:10:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 12:10:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 00:10:54.585328 16783 mini_ranger.cc:160] Time spent starting Ranger: real 9.836s user 0.003s sys 0.010s
Aug 12, 2026 12:11:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 12:11:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 12:11:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 12:11:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 12:11:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 00:11:19.518887 16783 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/rangerkms_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/rangerkms_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.99.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.99.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 00:11:19 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493479, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 00:11:19.632470 16783 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 00:11:19.644744 16783 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 00:11:20.734609 16783 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 00:11:20.734706 16783 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms
I20260812 00:11:20.734730 16783 mini_ranger_kms.cc:227] Using postgres at 127.16.99.212:48921
2026-08-12 00:11:21,110 [I] DB FLAVOR :POSTGRES
2026-08-12 00:11:21,110 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:11:21,110 [I] Checking connection
2026-08-12 00:11:21,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 00:11:21,411 [I] connection success
2026-08-12 00:11:21,411 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 00:11:21,411 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 00:11:21,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 00:11:21,665 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 00:11:21,666 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:11:21,666 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 00:11:21,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:48921/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 00:11:21,978 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 00:11:22.553263 16783 mini_ranger_kms.cc:310] Created kms service
I20260812 00:11:22.737931 16783 mini_ranger_kms.cc:326] Created kudu user
I20260812 00:11:22.788130 16783 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 00:11:22 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493479, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:11:23.264176 16783 mini_ranger_kms.cc:384] Added ranger policy
I20260812 00:11:23.264325 16783 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 00:11:23.264351 16783 mini_ranger_kms.cc:240] Using host: 127.16.99.212
I20260812 00:11:23.268858 16783 mini_ranger_kms.cc:276] Ranger KMS PID: 28611
I20260812 00:11:23.269019 16783 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.99.212:53099
Aug 12, 2026 12:11:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:11:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:11:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 12:11:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 12:11:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 12:11:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 00:11:25.714752 16783 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.082s user 0.000s sys 0.303s
I20260812 00:11:25.714880 16783 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 00:11:25.714978 16783 mini_ranger_kms.cc:401] 127.16.99.212:53099/kms/v1/keys
Aug 12 00:11:26 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.99.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 00:11:26 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.99.212@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
Aug 12 00:11:26 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493486, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.99.212@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
Aug 12 00:11:29 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:11:29.397825 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42305
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:42305
--ranger_config_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:53099/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:29.496050 28664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:29.496312 28664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:29.496338 28664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:29.497634 28664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:11:29.497694 28664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:29.497709 28664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:29.497722 28664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:11:29.497735 28664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:11:29.499541 28664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.99.212:53099/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:42305
--ranger_config_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:11:29.499768 28664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:29.499980 28664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:29.505509 28670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:29.505524 28669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:29.505555 28672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:29.505559 28664 server_base.cc:1034] running on GCE node
I20260812 00:11:29.505879 28664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:29.506210 28664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:29.507380 28664 hybrid_clock.cc:648] HybridClock initialized: now 1786493489507369 us; error 50 us; skew 500 ppm
Aug 12 00:11:29 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493489, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:29.517234 28664 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:11:29.517889 28664 webserver.cc:460] Webserver started at http://127.16.99.254:41781/ using document root <none> and password file <none>
I20260812 00:11:29.518167 28664 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:29.518239 28664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:29.518450 28664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:11:29 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493489, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:11:29.579627 28664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "98c092aec2ef4644a0d1f4bc28135726"
format_stamp: "Formatted at 2026-08-12 00:11:29 on dist-test-slave-zpfg"
server_key: "8ecd99ccf42c080bec650f63350e8933"
server_key_iv: "543ba62c9caf366d12080f4917b87abd"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:29.580070 28664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "98c092aec2ef4644a0d1f4bc28135726"
format_stamp: "Formatted at 2026-08-12 00:11:29 on dist-test-slave-zpfg"
server_key: "8ecd99ccf42c080bec650f63350e8933"
server_key_iv: "543ba62c9caf366d12080f4917b87abd"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:29.582122 28664 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 00:11:29.600075 28680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:29.600382 28664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:11:29.600521 28664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "98c092aec2ef4644a0d1f4bc28135726"
format_stamp: "Formatted at 2026-08-12 00:11:29 on dist-test-slave-zpfg"
server_key: "8ecd99ccf42c080bec650f63350e8933"
server_key_iv: "543ba62c9caf366d12080f4917b87abd"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:29.600694 28664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:29.621583 28664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:29.624048 28664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:29.624290 28664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:29.630117 28664 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:42305
I20260812 00:11:29.630167 28733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:42305 every 8 connection(s)
I20260812 00:11:29.630851 28664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:11:29.631661 28734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:29.634243 28734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726: Bootstrap starting.
I20260812 00:11:29.634939 28734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:29.635313 28734 log.cc:824] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:29.636155 28734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726: No bootstrap required, opened a new log
I20260812 00:11:29.638473 28734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c092aec2ef4644a0d1f4bc28135726" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42305 } }
I20260812 00:11:29.638675 28734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:29.638728 28734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c092aec2ef4644a0d1f4bc28135726, State: Initialized, Role: FOLLOWER
I20260812 00:11:29.638903 28734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c092aec2ef4644a0d1f4bc28135726" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42305 } }
I20260812 00:11:29.638989 28734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:11:29.639084 28734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:11:29.639155 28734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:29.639222 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 28664
I20260812 00:11:29.639339 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:11:29.639542 16783 external_mini_cluster.cc:1428] Setting key a4e7b3e6de062221c64f25491f24a319
I20260812 00:11:29.640002 28734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c092aec2ef4644a0d1f4bc28135726" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42305 } }
I20260812 00:11:29.640142 28734 leader_election.cc:302] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c092aec2ef4644a0d1f4bc28135726; no voters:
I20260812 00:11:29.640367 28734 leader_election.cc:288] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:11:29.640399 28739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:29.640610 28739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [term 1 LEADER]: Becoming Leader. State: Replica: 98c092aec2ef4644a0d1f4bc28135726, State: Running, Role: LEADER
I20260812 00:11:29.640720 28739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c092aec2ef4644a0d1f4bc28135726" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42305 } }
I20260812 00:11:29.641167 28739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c092aec2ef4644a0d1f4bc28135726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c092aec2ef4644a0d1f4bc28135726" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42305 } } }
I20260812 00:11:29.641266 28739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:29.641184 28741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c092aec2ef4644a0d1f4bc28135726. Latest consensus state: current_term: 1 leader_uuid: "98c092aec2ef4644a0d1f4bc28135726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c092aec2ef4644a0d1f4bc28135726" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42305 } } }
I20260812 00:11:29.641335 28741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [sys.catalog]: This master's current role is: LEADER
I20260812 00:11:29.641602 28734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:11:29.643482 28734 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 00:11:29.644058 28744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:11:29.647740 28744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:11:29 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:11:29.659436 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:29.640343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.16.99.254:42305)
0812 00:11:29.640578 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:29.640592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:29.642646 (+ 2054us) server_negotiation.cc:407] Connection header received
0812 00:11:29.642808 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:29.642817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:29.642892 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:29.643068 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:29.643894 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:29.648732 (+ 4838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:29.650102 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:29.650915 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:29.651454 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:29.654496 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:29.654502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:29.654502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:29.654517 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:29.656989 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:29.657480 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:29.657485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:29.657485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:29.657617 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:29.657878 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:29.657878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:29.657879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:29.658259 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:29.658956 (+ 697us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:29.659035 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:11:29.659182 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260812 00:11:29.663110 28744 catalog_manager.cc:1347] Generated new cluster ID: 8be4011954824e409ace377cd19b873a
I20260812 00:11:29.663239 28744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:11:29.680704 28744 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:11:29.681567 28744 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:11:29.692925 28744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726: Generated new TSK 0
I20260812 00:11:29.693178 28744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:11:30.502460 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:53099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42305
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:30.598678 28782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:30.598948 28782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:30.598992 28782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:30.600421 28782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:30.600512 28782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:30.600566 28782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:11:30.602546 28782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.99.212:53099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:11:30.602869 28782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:30.603161 28782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:30.609423 28787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:30.609486 28790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:30.609486 28788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:30.609726 28782 server_base.cc:1034] running on GCE node
I20260812 00:11:30.610108 28782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:30.610515 28782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:30.611716 28782 hybrid_clock.cc:648] HybridClock initialized: now 1786493490611691 us; error 43 us; skew 500 ppm
Aug 12 00:11:30 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:30.615356 28782 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:11:30.615942 28782 webserver.cc:460] Webserver started at http://127.16.99.193:46421/ using document root <none> and password file <none>
I20260812 00:11:30.616227 28782 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:30.616309 28782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:30.616479 28782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:11:30 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:11:30.634545 28782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "4f23702e3ecd49a4962b6f7110143858"
format_stamp: "Formatted at 2026-08-12 00:11:30 on dist-test-slave-zpfg"
server_key: "01bc536fbda695a24c46a106bde93492"
server_key_iv: "73700683dacd37cb272f328eaa0f7644"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:30.635103 28782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "4f23702e3ecd49a4962b6f7110143858"
format_stamp: "Formatted at 2026-08-12 00:11:30 on dist-test-slave-zpfg"
server_key: "01bc536fbda695a24c46a106bde93492"
server_key_iv: "73700683dacd37cb272f328eaa0f7644"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:30.638250 28782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 00:11:30.666555 28797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:30.666893 28782 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260812 00:11:30.667057 28782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "4f23702e3ecd49a4962b6f7110143858"
format_stamp: "Formatted at 2026-08-12 00:11:30 on dist-test-slave-zpfg"
server_key: "01bc536fbda695a24c46a106bde93492"
server_key_iv: "73700683dacd37cb272f328eaa0f7644"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:30.667179 28782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:30.695842 28782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:30.698292 28782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:30.698527 28782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:30.698791 28782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:30.699144 28782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:30.699204 28782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:30.699250 28782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:30.699270 28782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:30.706151 28782 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:43903
I20260812 00:11:30.706195 28921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:43903 every 8 connection(s)
I20260812 00:11:30.706691 28782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:11:30.710421 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 28782
I20260812 00:11:30.710572 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:11:30.710786 16783 external_mini_cluster.cc:1428] Setting key 2b967945978cbf88666c8b2c97c31eb8
Aug 12 00:11:30 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:11:30.723220 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:30.711749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:56757 (local address 127.16.99.254:42305)
0812 00:11:30.711857 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:30.711860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:30.712276 (+ 416us) server_negotiation.cc:407] Connection header received
0812 00:11:30.713183 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:30.713184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:30.713201 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:30.713244 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:30.714306 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:30.715168 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:30.716236 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:30.716568 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:30.716644 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:30.719805 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:30.719807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:30.719807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:30.719816 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:30.721473 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:30.721957 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:30.721959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:30.721959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:30.722036 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:30.722394 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:30.722395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:30.722396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:30.722541 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:30.722600 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:30.723010 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 00:11:30.723109 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
I20260812 00:11:30.724077 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:30.711312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42305 (local address 127.16.99.193:56757)
0812 00:11:30.712125 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:30.712187 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:30.713037 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:30.713450 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:30.713455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:30.713817 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:30.714177 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:30.714185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:30.715245 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:30.715246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:30.716120 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:30.716123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:30.716722 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:30.716723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:30.716889 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:30.717564 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:30.717583 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:30.719682 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:30.721594 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:30.721602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:30.721605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:30.721871 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:30.722120 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:30.722120 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:30.722121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:30.722282 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:30.722654 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:30.722661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:30.722842 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 00:11:30.723623 (+ 781us) client_negotiation.cc:240] Negotiation successful
0812 00:11:30.723813 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":113,"threads_started":1}
I20260812 00:11:30.724887 28922 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42305
I20260812 00:11:30.725003 28922 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:30.725210 28922 heartbeater.cc:511] Master 127.16.99.254:42305 requested a full tablet report, sending...
I20260812 00:11:30.725821 28697 ts_manager.cc:194] Registered new tserver with Master: 4f23702e3ecd49a4962b6f7110143858 (127.16.99.193:43903)
I20260812 00:11:30.726557 28697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:56757
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:11:30.750669 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:53099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42305
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:30.883524 28929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:30.883785 28929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:30.883832 28929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:30.885171 28929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:30.885246 28929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:30.885296 28929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:11:30.887436 28929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.99.212:53099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:11:30.887768 28929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:30.888037 28929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 12 00:11:30 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 00:11:30.894552 28939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:30.894588 28938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:30.894721 28942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:30.894670 28929 server_base.cc:1034] running on GCE node
I20260812 00:11:30.895145 28929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:30.895488 28929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:30.896669 28929 hybrid_clock.cc:648] HybridClock initialized: now 1786493490896640 us; error 48 us; skew 500 ppm
Aug 12 00:11:30 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:30.899832 28929 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:11:30.900454 28929 webserver.cc:460] Webserver started at http://127.16.99.194:42271/ using document root <none> and password file <none>
I20260812 00:11:30.900808 28929 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:30.900894 28929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:30.901067 28929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:11:30 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:11:30.916083 28929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "3224f47fee2d4172ba390d9a210816dc"
format_stamp: "Formatted at 2026-08-12 00:11:30 on dist-test-slave-zpfg"
server_key: "b48480eec76d423651daa9dbb730d631"
server_key_iv: "2b21a9f9a2fe0815a07b6dceaa7d0d78"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:30.916559 28929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "3224f47fee2d4172ba390d9a210816dc"
format_stamp: "Formatted at 2026-08-12 00:11:30 on dist-test-slave-zpfg"
server_key: "b48480eec76d423651daa9dbb730d631"
server_key_iv: "2b21a9f9a2fe0815a07b6dceaa7d0d78"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:30.920044 28929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 00:11:30.938547 28950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:30.939827 28929 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260812 00:11:30.940053 28929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "3224f47fee2d4172ba390d9a210816dc"
format_stamp: "Formatted at 2026-08-12 00:11:30 on dist-test-slave-zpfg"
server_key: "b48480eec76d423651daa9dbb730d631"
server_key_iv: "2b21a9f9a2fe0815a07b6dceaa7d0d78"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:30.940368 28929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:30.973933 28929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:30.976984 28929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:30.977246 28929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:30.977598 28929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:30.978106 28929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:30.978201 28929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:30.978282 28929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:30.978382 28929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:30.987226 28929 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:36765
I20260812 00:11:30.987669 28929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:11:30.991014 29074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:36765 every 8 connection(s)
I20260812 00:11:30.992933 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 28929
I20260812 00:11:30.993070 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:11:30.993284 16783 external_mini_cluster.cc:1428] Setting key 9eaeaac4ed47681c7bf083f19d1afc1b
Aug 12 00:11:31 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:11:31.011685 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:30.988927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:40401 (local address 127.16.99.254:42305)
0812 00:11:30.988980 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:30.988983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:30.993903 (+ 4920us) server_negotiation.cc:407] Connection header received
0812 00:11:30.998583 (+ 4680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:30.998586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:30.998613 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:30.998653 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:30.999758 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.000684 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.002571 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.002974 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.003032 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.007555 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:31.007557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:31.007557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:31.007568 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:31.009669 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:31.010295 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:31.010297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:31.010298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:31.010412 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:31.010788 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:31.010789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:31.010789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:31.010954 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:31.011015 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.011457 (+ 442us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.011567 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:11:31.012669 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:30.988601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42305 (local address 127.16.99.194:40401)
0812 00:11:30.993731 (+ 5130us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:30.993801 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:30.994783 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:30.998933 (+ 4150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:30.998939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:30.999267 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:30.999599 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:30.999608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.000850 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.000851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.002443 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.002449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.003178 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.003178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.003351 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.005346 (+ 1995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:31.005360 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:31.007389 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:31.009799 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:31.009810 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:31.009814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:31.010177 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:31.010501 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:31.010502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:31.010502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:31.010676 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:31.011059 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:31.011067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:31.011272 (+ 205us) client_negotiation.cc:769] Sending connection context
0812 00:11:31.012136 (+ 864us) client_negotiation.cc:240] Negotiation successful
0812 00:11:31.012377 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5056,"thread_start_us":94,"threads_started":1}
I20260812 00:11:31.013496 29075 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42305
I20260812 00:11:31.013607 29075 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:31.013839 29075 heartbeater.cc:511] Master 127.16.99.254:42305 requested a full tablet report, sending...
I20260812 00:11:31.014262 28697 ts_manager.cc:194] Registered new tserver with Master: 3224f47fee2d4172ba390d9a210816dc (127.16.99.194:36765)
I20260812 00:11:31.014868 28697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:40401
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:11:31.053005 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:53099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42305
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:11:31.227352 29082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:11:31.227636 29082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:11:31.227681 29082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:11:31.229139 29082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:11:31.229240 29082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:11:31.229290 29082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:11:31.231173 29082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.99.212:53099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:11:31.231478 29082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:11:31.231769 29082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:11:31.237593 29088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:31.237591 29087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:11:31.238390 29090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:11:31.238966 29082 server_base.cc:1034] running on GCE node
I20260812 00:11:31.239248 29082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:11:31.239550 29082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:11:31.240764 29082 hybrid_clock.cc:648] HybridClock initialized: now 1786493491240746 us; error 29 us; skew 500 ppm
Aug 12 00:11:31 dist-test-slave-zpfg krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493491, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:11:31.244112 29082 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:11:31.244807 29082 webserver.cc:460] Webserver started at http://127.16.99.195:41571/ using document root <none> and password file <none>
I20260812 00:11:31.245122 29082 fs_manager.cc:359] Metadata directory not provided
I20260812 00:11:31.245198 29082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:11:31.245410 29082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:11:31 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493491, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:11:31.261837 29082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "b973c7b393d44f82bdf8b40ded7fc123"
format_stamp: "Formatted at 2026-08-12 00:11:31 on dist-test-slave-zpfg"
server_key: "36782f9ac24d0e3e2511273de8ad3051"
server_key_iv: "b689c2ab0b1304a2ed5fbcd84ba258f6"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:31.271061 29082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "b973c7b393d44f82bdf8b40ded7fc123"
format_stamp: "Formatted at 2026-08-12 00:11:31 on dist-test-slave-zpfg"
server_key: "36782f9ac24d0e3e2511273de8ad3051"
server_key_iv: "b689c2ab0b1304a2ed5fbcd84ba258f6"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:31.282095 29082 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.000s
I20260812 00:11:31.322712 29097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:31.323045 29082 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260812 00:11:31.323203 29082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "b973c7b393d44f82bdf8b40ded7fc123"
format_stamp: "Formatted at 2026-08-12 00:11:31 on dist-test-slave-zpfg"
server_key: "36782f9ac24d0e3e2511273de8ad3051"
server_key_iv: "b689c2ab0b1304a2ed5fbcd84ba258f6"
server_key_version: "kuduclusterkey@0"
I20260812 00:11:31.323319 29082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:11:31.352211 29082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:11:31.356526 29082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:11:31.356814 29082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:11:31.357132 29082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:11:31.357595 29082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:11:31.357697 29082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:31.357793 29082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:11:31.357859 29082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:11:31.368330 29082 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:33833
I20260812 00:11:31.368942 29082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:11:31.370132 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 29082
I20260812 00:11:31.370270 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:11:31.370507 16783 external_mini_cluster.cc:1428] Setting key 1c5205b0e86724140f3b0d17c2871a7b
I20260812 00:11:31.377996 29215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:33833 every 8 connection(s)
Aug 12 00:11:31 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493491, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:11:31.405633 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.378872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42305 (local address 127.16.99.195:47665)
0812 00:11:31.385955 (+ 7083us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:31.386045 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:31.386805 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:31.387224 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:31.387229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:31.387533 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:11:31.387858 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.387866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.388900 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.388901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.390132 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.390135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.399136 (+ 9001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.399137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.399342 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.399993 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:11:31.400005 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:11:31.401746 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:11:31.403804 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:31.403815 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:31.403817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:31.404160 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:31.404384 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:11:31.404384 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:11:31.404385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:11:31.404521 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:11:31.404894 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:11:31.404902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:11:31.405103 (+ 201us) client_negotiation.cc:769] Sending connection context
0812 00:11:31.405211 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:11:31.405385 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7010,"thread_start_us":128,"threads_started":1}
I20260812 00:11:31.406258 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.378851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:47665 (local address 127.16.99.254:42305)
0812 00:11:31.379184 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.379194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.386182 (+ 6988us) server_negotiation.cc:407] Connection header received
0812 00:11:31.386951 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.386952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.386965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.386999 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:31.387984 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.388766 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.398567 (+ 9801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.398926 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.398974 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.401923 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:31.401924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:31.401925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:31.401934 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:31.403689 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:31.404252 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:31.404254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:31.404255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:31.404320 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:31.404588 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:31.404589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:31.404589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:31.404799 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:31.404862 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.406084 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.406171 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260812 00:11:31.406795 29216 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42305
I20260812 00:11:31.406920 29216 heartbeater.cc:464] Registering TS with master...
I20260812 00:11:31.407192 29216 heartbeater.cc:511] Master 127.16.99.254:42305 requested a full tablet report, sending...
I20260812 00:11:31.407611 28697 ts_manager.cc:194] Registered new tserver with Master: b973c7b393d44f82bdf8b40ded7fc123 (127.16.99.195:33833)
I20260812 00:11:31.408097 28697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:47665
I20260812 00:11:31.409551 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:11:31.430495 28926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.411477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.16.99.254:42305)
0812 00:11:31.411518 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.411520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.418578 (+ 7058us) server_negotiation.cc:407] Connection header received
0812 00:11:31.418705 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.418706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.418718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.418748 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:11:31.419535 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.420314 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.424926 (+ 4612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.425208 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.425286 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.427090 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:11:31.427092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:11:31.427092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:11:31.427100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:11:31.429057 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:31.429532 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:31.429534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:31.429534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:31.429600 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:11:31.429804 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:11:31.429805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:11:31.429805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:11:31.429948 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:11:31.430033 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.430275 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.430390 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:11:31.433665 28697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:11:31.436370 28697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:11:31.467424 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.453998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:33833 (local address 127.0.0.1:59652)
0812 00:11:31.455636 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:31.455691 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:31.455760 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:31.456224 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:31.456227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:31.456230 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:31.456323 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.456326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.458013 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.458014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.459412 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.459416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.467020 (+ 7604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.467022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.467144 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.467164 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 00:11:31.467254 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:11:31.467322 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"thread_start_us":97,"threads_started":1}
I20260812 00:11:31.467974 29232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.454307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43903 (local address 127.0.0.1:42866)
0812 00:11:31.458444 (+ 4137us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:31.458457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:31.458516 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:31.458851 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:31.458852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:31.458854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:31.458900 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.458902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.464994 (+ 6092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.464996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.466116 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.466121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.467749 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.467750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.467814 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.467818 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:31.467862 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 00:11:31.467903 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4116,"thread_start_us":127,"threads_started":1}
I20260812 00:11:31.468128 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.454125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.16.99.195:33833)
0812 00:11:31.455927 (+ 1802us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.455933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.455943 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:31.455986 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.455988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.456006 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.456114 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:31.456417 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.457916 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.460135 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.465723 (+ 5588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.465811 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.467938 (+ 2127us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.467982 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.468038 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1757,"thread_start_us":92,"threads_started":1}
I20260812 00:11:31.469211 29150 tablet_service.cc:1467] Processing CreateTablet for tablet d1236767364b4957821a1b91546252c7 (DEFAULT_TABLE table=test-table [id=a515de1f9d84457999fb974b4f8e83b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:31.469653 29150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1236767364b4957821a1b91546252c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:31.470860 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.454631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.16.99.193:43903)
0812 00:11:31.458633 (+ 4002us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.458638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.458648 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:11:31.458684 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.458686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.458702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.458775 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:31.463772 (+ 4997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.464869 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.466849 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.467584 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.467652 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.467711 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.470699 (+ 2988us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.470769 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3961,"thread_start_us":58,"threads_started":1}
I20260812 00:11:31.472815 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.454917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:36765 (local address 127.0.0.1:37300)
0812 00:11:31.466276 (+ 11359us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:31.466288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:31.466331 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:31.466720 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:31.466722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:31.466724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:31.466769 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.466770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.469466 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.469467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.470567 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.470571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.472567 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.472568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.472634 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.472639 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:11:31.472690 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:11:31.472734 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11337,"thread_start_us":87,"threads_started":1}
I20260812 00:11:31.474246 28850 tablet_service.cc:1467] Processing CreateTablet for tablet d1236767364b4957821a1b91546252c7 (DEFAULT_TABLE table=test-table [id=a515de1f9d84457999fb974b4f8e83b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:31.474684 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1236767364b4957821a1b91546252c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:31.478413 29237 tablet_bootstrap.cc:492] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858: Bootstrap starting.
I20260812 00:11:31.479055 29236 tablet_bootstrap.cc:492] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Bootstrap starting.
I20260812 00:11:31.479364 29237 tablet_bootstrap.cc:654] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:31.479825 29237 log.cc:824] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:31.480362 29236 tablet_bootstrap.cc:654] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:31.480784 29236 log.cc:824] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:31.480839 29237 tablet_bootstrap.cc:492] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858: No bootstrap required, opened a new log
I20260812 00:11:31.480908 29237 ts_tablet_manager.cc:1397] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 00:11:31.483510 29237 raft_consensus.cc:348] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.483690 29237 raft_consensus.cc:374] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:31.483721 29237 raft_consensus.cc:729] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f23702e3ecd49a4962b6f7110143858, State: Initialized, Role: FOLLOWER
I20260812 00:11:31.484030 29237 consensus_queue.cc:260] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.484421 29237 ts_tablet_manager.cc:1428] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260812 00:11:31.484586 28922 heartbeater.cc:503] Master 127.16.99.254:42305 was elected leader, sending a full tablet report...
I20260812 00:11:31.486267 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.455207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.16.99.194:36765)
0812 00:11:31.466470 (+ 11263us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.466476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.466487 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:11:31.466533 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.466535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.466552 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.466622 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:31.468258 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.469362 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.470990 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.472375 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.472452 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.472527 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.473030 (+ 503us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.486157 (+ 13127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11222,"thread_start_us":62,"threads_started":1}
I20260812 00:11:31.487195 29002 tablet_service.cc:1467] Processing CreateTablet for tablet d1236767364b4957821a1b91546252c7 (DEFAULT_TABLE table=test-table [id=a515de1f9d84457999fb974b4f8e83b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:11:31.487569 29002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1236767364b4957821a1b91546252c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:11:31.491294 29241 tablet_bootstrap.cc:492] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Bootstrap starting.
I20260812 00:11:31.491568 29236 tablet_bootstrap.cc:492] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: No bootstrap required, opened a new log
I20260812 00:11:31.491676 29236 ts_tablet_manager.cc:1397] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260812 00:11:31.492362 29241 tablet_bootstrap.cc:654] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Neither blocks nor log segments found. Creating new log.
I20260812 00:11:31.492707 29241 log.cc:824] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Log is configured to *not* fsync() on all Append() calls
I20260812 00:11:31.493963 29236 raft_consensus.cc:348] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.494144 29236 raft_consensus.cc:374] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:31.494177 29236 raft_consensus.cc:729] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b973c7b393d44f82bdf8b40ded7fc123, State: Initialized, Role: FOLLOWER
I20260812 00:11:31.494436 29236 consensus_queue.cc:260] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.494802 29236 ts_tablet_manager.cc:1428] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:11:31.494957 29216 heartbeater.cc:503] Master 127.16.99.254:42305 was elected leader, sending a full tablet report...
I20260812 00:11:31.496737 29241 tablet_bootstrap.cc:492] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: No bootstrap required, opened a new log
I20260812 00:11:31.496863 29241 ts_tablet_manager.cc:1397] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 00:11:31.498919 29241 raft_consensus.cc:348] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.499166 29241 raft_consensus.cc:374] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:11:31.499253 29241 raft_consensus.cc:729] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3224f47fee2d4172ba390d9a210816dc, State: Initialized, Role: FOLLOWER
I20260812 00:11:31.499444 29241 consensus_queue.cc:260] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.499804 29241 ts_tablet_manager.cc:1428] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:11:31.499989 29075 heartbeater.cc:503] Master 127.16.99.254:42305 was elected leader, sending a full tablet report...
Aug 12 00:11:31 dist-test-slave-zpfg krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:11:31.528133 29243 raft_consensus.cc:482] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:11:31.528942 29243 raft_consensus.cc:504] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.531162 29243 leader_election.cc:288] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f23702e3ecd49a4962b6f7110143858 (127.16.99.193:43903), 3224f47fee2d4172ba390d9a210816dc (127.16.99.194:36765)
I20260812 00:11:31.533164 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.529679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43903 (local address 127.16.99.195:36343)
0812 00:11:31.529735 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:31.529756 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:31.529805 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:31.529989 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:31.529990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:31.529992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:31.530066 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.530068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.530971 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.530972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.532062 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.532065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.532897 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.532899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.532981 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.532985 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:11:31.533036 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:11:31.533088 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:11:31.533954 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.529738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:36343 (local address 127.16.99.193:43903)
0812 00:11:31.529787 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.529789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.529800 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:11:31.529875 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.529875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.529885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.529936 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:31.530147 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.530870 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.532180 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.532774 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.532851 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.532861 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.533828 (+ 967us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.533879 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:11:31.534271 28873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1236767364b4957821a1b91546252c7" candidate_uuid: "b973c7b393d44f82bdf8b40ded7fc123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23702e3ecd49a4962b6f7110143858" is_pre_election: true
I20260812 00:11:31.534487 28873 raft_consensus.cc:2454] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b973c7b393d44f82bdf8b40ded7fc123 in term 0.
I20260812 00:11:31.535324 29098 leader_election.cc:302] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f23702e3ecd49a4962b6f7110143858, b973c7b393d44f82bdf8b40ded7fc123; no voters:
I20260812 00:11:31.537597 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.530893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:36765 (local address 127.16.99.195:60211)
0812 00:11:31.533253 (+ 2360us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:31.533266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:11:31.533314 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:11:31.534253 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:11:31.534254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:11:31.534256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:11:31.534297 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.534299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.535607 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.535608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.536614 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:11:31.536618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.537374 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.537375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:11:31.537439 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.537442 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:11:31.537491 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:11:31.537529 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2335,"thread_start_us":101,"threads_started":1}
I20260812 00:11:31.538049 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.531101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:60211 (local address 127.16.99.194:36765)
0812 00:11:31.531144 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.531146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.533452 (+ 2306us) server_negotiation.cc:407] Connection header received
0812 00:11:31.533486 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.533487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.533501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.533551 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:11:31.534411 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.535044 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.536716 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.537235 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.537328 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.537339 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.537935 (+ 596us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.537985 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:11:31.538612 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1236767364b4957821a1b91546252c7" candidate_uuid: "b973c7b393d44f82bdf8b40ded7fc123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3224f47fee2d4172ba390d9a210816dc" is_pre_election: true
I20260812 00:11:31.538782 29007 raft_consensus.cc:2454] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b973c7b393d44f82bdf8b40ded7fc123 in term 0.
I20260812 00:11:31.539148 29243 raft_consensus.cc:2781] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:11:31.539283 29243 raft_consensus.cc:482] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:11:31.539373 29243 raft_consensus.cc:3037] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:31.540475 29243 raft_consensus.cc:504] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.540879 28873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1236767364b4957821a1b91546252c7" candidate_uuid: "b973c7b393d44f82bdf8b40ded7fc123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f23702e3ecd49a4962b6f7110143858"
I20260812 00:11:31.540984 28873 raft_consensus.cc:3037] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:31.540998 29007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1236767364b4957821a1b91546252c7" candidate_uuid: "b973c7b393d44f82bdf8b40ded7fc123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3224f47fee2d4172ba390d9a210816dc"
I20260812 00:11:31.541066 29007 raft_consensus.cc:3037] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:11:31.541955 28873 raft_consensus.cc:2454] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b973c7b393d44f82bdf8b40ded7fc123 in term 1.
I20260812 00:11:31.542099 29007 raft_consensus.cc:2454] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b973c7b393d44f82bdf8b40ded7fc123 in term 1.
I20260812 00:11:31.542402 29099 leader_election.cc:302] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3224f47fee2d4172ba390d9a210816dc, b973c7b393d44f82bdf8b40ded7fc123; no voters:
I20260812 00:11:31.542797 29243 leader_election.cc:288] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [CANDIDATE]: Term 1 election: Requested vote from peers 4f23702e3ecd49a4962b6f7110143858 (127.16.99.193:43903), 3224f47fee2d4172ba390d9a210816dc (127.16.99.194:36765)
I20260812 00:11:31.542879 29243 raft_consensus.cc:2781] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:11:31.542960 29243 raft_consensus.cc:686] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 1 LEADER]: Becoming Leader. State: Replica: b973c7b393d44f82bdf8b40ded7fc123, State: Running, Role: LEADER
I20260812 00:11:31.543066 29243 consensus_queue.cc:237] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } }
I20260812 00:11:31.544023 28697 catalog_manager.cc:5495] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 reported cstate change: term changed from 0 to 1, leader changed from <none> to b973c7b393d44f82bdf8b40ded7fc123 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "b973c7b393d44f82bdf8b40ded7fc123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b973c7b393d44f82bdf8b40ded7fc123" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 } health_report { overall_health: UNKNOWN } } }
I20260812 00:11:31.556978 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.553473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.16.99.195:33833)
0812 00:11:31.553522 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:11:31.553524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:11:31.553532 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:11:31.553574 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:11:31.553574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:11:31.553596 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:11:31.553674 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:11:31.553875 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.554609 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.555705 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:11:31.555994 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:11:31.556056 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:11:31.556287 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:11:31.556424 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:11:31.556524 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 00:11:31.556831 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 00:11:31.556856 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:11:31.563648 29007 raft_consensus.cc:1261] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 1 FOLLOWER]: Refusing update from remote peer b973c7b393d44f82bdf8b40ded7fc123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:31.564095 28873 raft_consensus.cc:1261] T d1236767364b4957821a1b91546252c7 P 4f23702e3ecd49a4962b6f7110143858 [term 1 FOLLOWER]: Refusing update from remote peer b973c7b393d44f82bdf8b40ded7fc123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:11:31.564138 29246 consensus_queue.cc:1035] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3224f47fee2d4172ba390d9a210816dc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 36765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:31.564481 29246 consensus_queue.cc:1035] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f23702e3ecd49a4962b6f7110143858" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:11:31.577663 29249 mvcc.cc:205] Tried to move back new op lower bound from 7317477341440880640 to 7317477341361348608. Current Snapshot: MvccSnapshot[applied={T|T < 7317477341440880640}]
I20260812 00:11:31.577895 29250 mvcc.cc:205] Tried to move back new op lower bound from 7317477341440880640 to 7317477341361348608. Current Snapshot: MvccSnapshot[applied={T|T < 7317477341440880640}]
I20260812 00:11:31.581410 29252 mvcc.cc:205] Tried to move back new op lower bound from 7317477341440880640 to 7317477341361348608. Current Snapshot: MvccSnapshot[applied={T|T < 7317477341440880640}]
I20260812 00:11:31.584951 28696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:31.585075 28696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:11:31.586063 28696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98c092aec2ef4644a0d1f4bc28135726: Sending DeleteTablet for 3 replicas of tablet d1236767364b4957821a1b91546252c7
I20260812 00:11:31.586519 28850 tablet_service.cc:1514] Processing DeleteTablet for tablet d1236767364b4957821a1b91546252c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:31 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:42866
I20260812 00:11:31.586750 29150 tablet_service.cc:1514] Processing DeleteTablet for tablet d1236767364b4957821a1b91546252c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:31 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:59652
I20260812 00:11:31.587165 29002 tablet_service.cc:1514] Processing DeleteTablet for tablet d1236767364b4957821a1b91546252c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:11:31 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:37300
I20260812 00:11:31.587903 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 28782
I20260812 00:11:31.592533 29261 tablet_replica.cc:330] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: stopping tablet replica
I20260812 00:11:31.592758 29261 raft_consensus.cc:2229] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:11:31.593022 29261 raft_consensus.cc:2258] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:31.593696 29261 ts_tablet_manager.cc:1905] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:31.600274 29260 tablet_replica.cc:330] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: stopping tablet replica
I20260812 00:11:31.600531 29260 raft_consensus.cc:2229] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:11:31.600804 29260 raft_consensus.cc:2258] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:11:31.601125 29261 ts_tablet_manager.cc:1918] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:11:31.601691 29260 ts_tablet_manager.cc:1905] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:11:31.603539 29261 log.cc:1196] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/d1236767364b4957821a1b91546252c7
I20260812 00:11:31.603973 29261 ts_tablet_manager.cc:1939] T d1236767364b4957821a1b91546252c7 P 3224f47fee2d4172ba390d9a210816dc: Deleting consensus metadata
I20260812 00:11:31.604645 28682 catalog_manager.cc:4841] TS 3224f47fee2d4172ba390d9a210816dc (127.16.99.194:36765): tablet d1236767364b4957821a1b91546252c7 (table test-table [id=a515de1f9d84457999fb974b4f8e83b0]) successfully deleted
I20260812 00:11:31.604935 29260 ts_tablet_manager.cc:1918] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:11:31.605104 29260 log.cc:1196] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/d1236767364b4957821a1b91546252c7
I20260812 00:11:31.605620 29260 ts_tablet_manager.cc:1939] T d1236767364b4957821a1b91546252c7 P b973c7b393d44f82bdf8b40ded7fc123: Deleting consensus metadata
I20260812 00:11:31.606205 28683 catalog_manager.cc:4841] TS b973c7b393d44f82bdf8b40ded7fc123 (127.16.99.195:33833): tablet d1236767364b4957821a1b91546252c7 (table test-table [id=a515de1f9d84457999fb974b4f8e83b0]) successfully deleted
W20260812 00:11:31.606561 28681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:43903 (error 108)
I20260812 00:11:31.607687 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:11:31.607533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:43903 (local address 127.0.0.1:42878)
0812 00:11:31.607592 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 00:11:31.607650 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:43903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 00:11:31.607698 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 28929
W20260812 00:11:31.612877 28681 catalog_manager.cc:4584] TS 4f23702e3ecd49a4962b6f7110143858 (127.16.99.193:43903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1236767364b4957821a1b91546252c7: Network error: Client connection negotiation failed: client connection to 127.16.99.193:43903: connect: Connection refused (error 111)
I20260812 00:11:31.618646 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 29082
I20260812 00:11:31.624734 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 28664
I20260812 00:11:31.632314 16783 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 00:11:31.977680 16783 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 00:11:31.977880 16783 mini_ranger.cc:66] Stopping Ranger...
I20260812 00:11:32.333451 16783 mini_ranger.cc:68] Stopped Ranger
2026-08-12 00:11:32.333 UTC [28228] LOG: received smart shutdown request
2026-08-12 00:11:32.338 UTC [28228] LOG: background worker "logical replication launcher" (PID 28238) exited with exit code 1
2026-08-12 00:11:32.338 UTC [28232] LOG: shutting down
2026-08-12 00:11:32.559 UTC [28228] LOG: database system is shut down
2026-08-12T00:11:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53231 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:11:32 dist-test-slave-zpfg krb5kdc[29274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:11:32 dist-test-slave-zpfg krb5kdc[29274](info): set up 2 sockets
Aug 12 00:11:32 dist-test-slave-zpfg krb5kdc[29274](info): commencing operation
krb5kdc: starting...
W20260812 00:11:33.838660 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:11:33 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:11:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/rangeradmin_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/rangeradmin_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/rangerlookup_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/rangerlookup_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
I20260812 00:11:34.037788 16783 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcVEJCj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/postgres -l logfile start
2026-08-12 00:11:36.957 UTC [29305] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 00:11:36.957 UTC [29305] LOG: listening on IPv4 address "127.16.99.212", port 41953
2026-08-12 00:11:36.962 UTC [29305] LOG: listening on Unix socket "/tmp/.s.PGSQL.41953"
2026-08-12 00:11:37.004 UTC [29308] LOG: database system was shut down at 2026-08-12 00:11:35 UTC
2026-08-12 00:11:37.009 UTC [29305] LOG: database system is ready to accept connections
I20260812 00:11:38.092242 16783 mini_postgres.cc:96] Postgres bound to 41953
2026-08-12 00:11:38.100 UTC [29318] FATAL: database "slave" does not exist
127.16.99.212:41953 - accepting connections
I20260812 00:11:38.100795 16783 mini_ranger.cc:161] Starting Ranger...
I20260812 00:11:38.117944 16783 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 00:11:38.726933 16783 mini_ranger.cc:87] Created ranger Postgres database
I20260812 00:11:38.727031 16783 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-admin
2026-08-12 00:11:39,036 [I] DB FLAVOR :POSTGRES
2026-08-12 00:11:39,037 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:11:39,037 [I] Checking connection..
2026-08-12 00:11:39,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 00:11:39,301 [I] Checking connection passed.
2026-08-12 00:11:39,302 [I] --------- Verifying version history table ---------
2026-08-12 00:11:39,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:11:39,574 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:11:39,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:11:39,854 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 00:11:39,854 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 00:11:39,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 00:11:40.104 UTC [29385] WARNING: there is no transaction in progress
2026-08-12 00:11:40,113 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 00:11:40,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 00:11:40,367 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 00:11:40,367 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:11:40,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 00:11:40,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 00:11:40,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpfg.c.gcp-upstream.internal','N') ;"
2026-08-12 00:11:41,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:11:41,488 [I] Table x_portal_user does not exist in database ranger
2026-08-12 00:11:41,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:11:41,791 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 00:11:41,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:11:42,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:11:42,348 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 00:11:42,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 00:11:43.466 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.488 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.509 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.529 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.552 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.575 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.645 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.654 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.672 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.684 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.698 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.715 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.725 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.733 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.742 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:43.751 UTC [29498] WARNING: there is no transaction in progress
2026-08-12 00:11:44,554 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 00:11:44,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 00:11:44,839 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 00:11:44,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 00:11:45,103 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 00:11:45,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 00:11:45,413 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 00:11:45,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 00:11:45,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 00:11:45,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpfg.c.gcp-upstream.internal';"
2026-08-12 00:11:46,243 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 00:11:46,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 00:11:46,515 [I] DB_PATCHES have already been applied
I20260812 00:11:46.522014 16783 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcVEJCj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 00:11:46.522089 16783 mini_ranger.cc:193] Using host: 127.16.99.212
I20260812 00:11:46.526453 16783 mini_ranger.cc:227] Ranger admin URL: http://127.16.99.212:39853
Aug 12, 2026 12:11:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:11:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:11:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 12:11:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 12:11:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 12:11:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 12:11:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 00:11:47.784713 16783 mini_ranger.cc:160] Time spent starting Ranger: real 9.684s user 0.000s sys 0.012s
Aug 12, 2026 12:12:09 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 12:12:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 12:12:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 12:12:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 12:12:10 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 00:12:11.321296 16783 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.99.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.99.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/rangerkms_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.99.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/rangerkms_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.99.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.99.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.99.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/HTTP_127.16.99.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 00:12:11 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493531, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 00:12:11.416419 16783 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 00:12:11.427495 16783 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 00:12:12.470758 16783 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 00:12:12.470860 16783 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms
I20260812 00:12:12.470882 16783 mini_ranger_kms.cc:227] Using postgres at 127.16.99.212:41953
2026-08-12 00:12:12,651 [I] DB FLAVOR :POSTGRES
2026-08-12 00:12:12,651 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 00:12:12,652 [I] Checking connection
2026-08-12 00:12:12,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 00:12:12,927 [I] connection success
2026-08-12 00:12:12,928 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 00:12:12,928 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 00:12:12,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 00:12:13,234 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 00:12:13,235 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 00:12:13,235 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 00:12:13,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.99.212:41953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 00:12:13,565 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 00:12:14.166214 16783 mini_ranger_kms.cc:310] Created kms service
I20260812 00:12:14.357424 16783 mini_ranger_kms.cc:326] Created kudu user
I20260812 00:12:14.405529 16783 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 00:12:14 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493531, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:14.899608 16783 mini_ranger_kms.cc:384] Added ranger policy
I20260812 00:12:14.899857 16783 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcVEJCj/build/release/bin/postgresql.jar:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcVEJCj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 00:12:14.899884 16783 mini_ranger_kms.cc:240] Using host: 127.16.99.212
I20260812 00:12:14.904202 16783 mini_ranger_kms.cc:276] Ranger KMS PID: 29688
I20260812 00:12:14.904311 16783 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.99.212:52659
Aug 12, 2026 12:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 12:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 12:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 12:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 12:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 12:12:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 00:12:17.353703 16783 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.937s user 0.007s sys 0.158s
I20260812 00:12:17.353804 16783 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 00:12:17.353853 16783 mini_ranger_kms.cc:401] 127.16.99.212:52659/kms/v1/keys
Aug 12 00:12:17 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493537, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.99.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 00:12:18 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493537, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.99.212@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
Aug 12 00:12:18 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493537, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.99.212@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
Aug 12 00:12:20 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:12:21.034744 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:35409
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:35409
--ranger_config_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:21.124413 29742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:21.124655 29742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:21.124678 29742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:21.125963 29742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:21.126021 29742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:21.126036 29742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:21.126049 29742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:21.126061 29742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:21.127851 29742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.99.212:52659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:35409
--ranger_config_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:21.128159 29742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:21.128441 29742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:21.134665 29749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:21.134717 29748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:21.134759 29742 server_base.cc:1034] running on GCE node
W20260812 00:12:21.134717 29751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:21.135370 29742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:21.135674 29742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:21.136850 29742 hybrid_clock.cc:648] HybridClock initialized: now 1786493541136822 us; error 42 us; skew 500 ppm
Aug 12 00:12:21 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493541, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:21.146193 29742 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:21.146849 29742 webserver.cc:460] Webserver started at http://127.16.99.254:46483/ using document root <none> and password file <none>
I20260812 00:12:21.147118 29742 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:21.147166 29742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:21.147341 29742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:12:21 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493541, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:21.200254 29742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "d504710524cf47cb9ec93d9bbdf9316e"
format_stamp: "Formatted at 2026-08-12 00:12:21 on dist-test-slave-zpfg"
server_key: "43cb7fc1aaf32020e9e3caf3400e3196"
server_key_iv: "4f35995e46e71266d70eddf90da731f0"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:21.200678 29742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "d504710524cf47cb9ec93d9bbdf9316e"
format_stamp: "Formatted at 2026-08-12 00:12:21 on dist-test-slave-zpfg"
server_key: "43cb7fc1aaf32020e9e3caf3400e3196"
server_key_iv: "4f35995e46e71266d70eddf90da731f0"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:21.202517 29742 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260812 00:12:21.219734 29758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:21.219940 29742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:12:21.220047 29742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "d504710524cf47cb9ec93d9bbdf9316e"
format_stamp: "Formatted at 2026-08-12 00:12:21 on dist-test-slave-zpfg"
server_key: "43cb7fc1aaf32020e9e3caf3400e3196"
server_key_iv: "4f35995e46e71266d70eddf90da731f0"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:21.220176 29742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:21.273033 29742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:21.275436 29742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:21.275588 29742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:21.280916 29742 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:35409
I20260812 00:12:21.280972 29815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:35409 every 8 connection(s)
I20260812 00:12:21.281451 29742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:21.282312 29816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:21.283713 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 29742
I20260812 00:12:21.283812 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:12:21.284025 16783 external_mini_cluster.cc:1428] Setting key 69e155eb80d90a0ac3c9e0d96a241bbc
I20260812 00:12:21.285444 29816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Bootstrap starting.
I20260812 00:12:21.286307 29816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:21.286726 29816 log.cc:824] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:21.287595 29816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: No bootstrap required, opened a new log
I20260812 00:12:21.290171 29816 raft_consensus.cc:348] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
I20260812 00:12:21.290400 29816 raft_consensus.cc:374] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:21.290477 29816 raft_consensus.cc:729] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d504710524cf47cb9ec93d9bbdf9316e, State: Initialized, Role: FOLLOWER
I20260812 00:12:21.290644 29816 consensus_queue.cc:260] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
I20260812 00:12:21.290722 29816 raft_consensus.cc:388] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:21.290746 29816 raft_consensus.cc:482] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:21.290823 29816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:21.291452 29816 raft_consensus.cc:504] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
I20260812 00:12:21.291550 29816 leader_election.cc:302] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d504710524cf47cb9ec93d9bbdf9316e; no voters:
I20260812 00:12:21.291796 29816 leader_election.cc:288] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:21.291877 29821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:21.292083 29821 raft_consensus.cc:686] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 LEADER]: Becoming Leader. State: Replica: d504710524cf47cb9ec93d9bbdf9316e, State: Running, Role: LEADER
I20260812 00:12:21.292138 29816 sys_catalog.cc:564] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:21.292193 29821 consensus_queue.cc:237] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
Aug 12 00:12:21 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:21.293355 29822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d504710524cf47cb9ec93d9bbdf9316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } } }
I20260812 00:12:21.293466 29822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:21.293727 29823 sys_catalog.cc:455] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d504710524cf47cb9ec93d9bbdf9316e. Latest consensus state: current_term: 1 leader_uuid: "d504710524cf47cb9ec93d9bbdf9316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } } }
I20260812 00:12:21.293804 29823 sys_catalog.cc:458] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:21.293936 29833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:21.294031 29816 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 00:12:21.296571 29833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:21.301209 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:21.284712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.16.99.254:35409)
0812 00:12:21.284927 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:21.284937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:21.284966 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:12:21.285623 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:21.285637 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:21.285705 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:21.285856 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:21.286881 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.288139 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.289334 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:21.289749 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:21.289859 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:21.292888 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:21.292893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:21.292893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:21.292908 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:21.298095 (+ 5187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:21.298810 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:21.298820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:21.298821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:21.299028 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:21.299391 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:21.299392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:21.299392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:21.300058 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:21.300313 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:21.300424 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 00:12:21.300834 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
I20260812 00:12:21.302193 29833 catalog_manager.cc:1347] Generated new cluster ID: 1e8cd6f2f5134db3b430227fd73cce20
I20260812 00:12:21.302275 29833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:21.321866 29833 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:21.322678 29833 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:21.328732 29833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Generated new TSK 0
I20260812 00:12:21.328972 29833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:12:22.119053 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:35409
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:22.218302 29864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:22.218631 29864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:22.218675 29864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:22.220022 29864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:22.220096 29864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:22.220145 29864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:12:22.221810 29864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.99.212:52659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:22.222108 29864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:22.222535 29864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:22.228353 29869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:22.228394 29872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:22.228374 29870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:22.229540 29864 server_base.cc:1034] running on GCE node
I20260812 00:12:22.229848 29864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:22.230191 29864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:22.231418 29864 hybrid_clock.cc:648] HybridClock initialized: now 1786493542231387 us; error 48 us; skew 500 ppm
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:22.234791 29864 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:12:22.235476 29864 webserver.cc:460] Webserver started at http://127.16.99.193:36979/ using document root <none> and password file <none>
I20260812 00:12:22.235807 29864 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:22.235864 29864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:22.236043 29864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:22.256820 29864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "a6f3a05c44634b70a4ed90e457959865"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "2c1284c145e896cedd6e0513c52c31cc"
server_key_iv: "145c4befbd1be5be844bd842c94f271f"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.257398 29864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "a6f3a05c44634b70a4ed90e457959865"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "2c1284c145e896cedd6e0513c52c31cc"
server_key_iv: "145c4befbd1be5be844bd842c94f271f"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.262167 29864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 00:12:22.279994 29879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.280210 29864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:22.280310 29864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "a6f3a05c44634b70a4ed90e457959865"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "2c1284c145e896cedd6e0513c52c31cc"
server_key_iv: "145c4befbd1be5be844bd842c94f271f"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.280421 29864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:22.309195 29864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:22.311856 29864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:22.312093 29864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:22.312357 29864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:22.312835 29864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:22.312902 29864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.312948 29864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:22.312978 29864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.320957 29864 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:46159
I20260812 00:12:22.321563 29864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:22.321854 30004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:46159 every 8 connection(s)
I20260812 00:12:22.329631 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 29864
I20260812 00:12:22.329734 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:22.329937 16783 external_mini_cluster.cc:1428] Setting key 0638aeeb6fc2bce4f7442f39ef061be6
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:22.335337 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.322843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:37555 (local address 127.16.99.254:35409)
0812 00:12:22.323039 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:22.323041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:22.323459 (+ 418us) server_negotiation.cc:407] Connection header received
0812 00:12:22.324146 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:22.324148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:22.324164 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:22.324238 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:22.325091 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.325749 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.327233 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.327614 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.327699 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.331621 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:22.331623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:22.331623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:22.331634 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:22.333647 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:22.334165 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:22.334167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:22.334167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:22.334237 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:22.334582 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:22.334583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:22.334583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:22.334723 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:22.334807 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:22.335109 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 00:12:22.335206 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260812 00:12:22.335800 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.322741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:35409 (local address 127.16.99.193:37555)
0812 00:12:22.323128 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:22.323184 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:22.323947 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:22.324404 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:22.324409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:22.324672 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:22.324901 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.324906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.325882 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.325883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.327115 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.327118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.327805 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.327807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.327990 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.329302 (+ 1312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:22.329314 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:22.331475 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:22.333766 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:22.333777 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:22.333780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:22.334052 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:22.334317 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:22.334318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:22.334319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:22.334500 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:22.334798 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:22.334806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:22.334962 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 00:12:22.335075 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:12:22.335258 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":109,"threads_started":1}
I20260812 00:12:22.336416 30005 heartbeater.cc:347] Connected to a master server at 127.16.99.254:35409
I20260812 00:12:22.336545 30005 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:22.336755 30005 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:22.337370 29778 ts_manager.cc:194] Registered new tserver with Master: a6f3a05c44634b70a4ed90e457959865 (127.16.99.193:46159)
I20260812 00:12:22.338069 29778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:37555
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:12:22.369980 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:35409
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:22.497243 30013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:22.497592 30013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:22.497640 30013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:22.499662 30013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:22.499751 30013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:22.499799 30013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:12:22.502420 30013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.99.212:52659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:22.502730 30013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:22.503023 30013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:22.510744 30022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:22.510939 30013 server_base.cc:1034] running on GCE node
W20260812 00:12:22.510771 30024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:22.510744 30021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:22.511569 30013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:22.511916 30013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:22.513098 30013 hybrid_clock.cc:648] HybridClock initialized: now 1786493542513063 us; error 49 us; skew 500 ppm
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:22.516618 30013 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:12:22.517211 30013 webserver.cc:460] Webserver started at http://127.16.99.194:33643/ using document root <none> and password file <none>
I20260812 00:12:22.517527 30013 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:22.517608 30013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:22.517752 30013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:22.534852 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "79ce408111cc4ace9a13544209f40d61"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "0828e8a8344663ebef5c55a6cbd93479"
server_key_iv: "4b591b27756fc6476ab2a62c7e8a9a3a"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.535364 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "79ce408111cc4ace9a13544209f40d61"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "0828e8a8344663ebef5c55a6cbd93479"
server_key_iv: "4b591b27756fc6476ab2a62c7e8a9a3a"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.540926 30013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 00:12:22.563472 30032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.563795 30013 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260812 00:12:22.563902 30013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "79ce408111cc4ace9a13544209f40d61"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "0828e8a8344663ebef5c55a6cbd93479"
server_key_iv: "4b591b27756fc6476ab2a62c7e8a9a3a"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.564003 30013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:22.579347 30013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:22.582072 30013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:22.582302 30013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:22.582708 30013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:22.583189 30013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:22.583284 30013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.583360 30013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:22.583422 30013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.590744 30013 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:44425
I20260812 00:12:22.591354 30013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:22.596282 30152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:44425 every 8 connection(s)
I20260812 00:12:22.599429 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30013
I20260812 00:12:22.599570 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:22.599781 16783 external_mini_cluster.cc:1428] Setting key 2202c2821e6c49c1c5767f8ce1f31e53
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:22.610582 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.597459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:58667 (local address 127.16.99.254:35409)
0812 00:12:22.597509 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:22.597511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:22.597772 (+ 261us) server_negotiation.cc:407] Connection header received
0812 00:12:22.598479 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:22.598480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:22.598500 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:22.598584 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:22.599578 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.600319 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.601898 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.602197 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.602249 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.605271 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:22.605273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:22.605273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:22.605284 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:22.607214 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:22.609125 (+ 1911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:22.609128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:22.609128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:22.609230 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:22.609617 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:22.609618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:22.609618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:22.609785 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:22.609834 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:22.610292 (+ 458us) server_negotiation.cc:299] Negotiation successful
0812 00:12:22.610454 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:12:22.611532 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.597119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:35409 (local address 127.16.99.194:58667)
0812 00:12:22.597582 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:22.597629 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:22.598297 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:22.598760 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:22.598765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:22.599078 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:22.599407 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.599436 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.600452 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.600453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.601748 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.601752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.602422 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.602423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.602595 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.603364 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:22.603377 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:22.605120 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:22.607344 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:22.607358 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:22.607389 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:22.607819 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:22.609328 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:22.609329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:22.609329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:22.609525 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:22.609893 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:22.609902 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:22.610111 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 00:12:22.611033 (+ 922us) client_negotiation.cc:240] Negotiation successful
0812 00:12:22.611256 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":126,"threads_started":1}
I20260812 00:12:22.612282 30153 heartbeater.cc:347] Connected to a master server at 127.16.99.254:35409
I20260812 00:12:22.612388 30153 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:22.612608 30153 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:22.613106 29778 ts_manager.cc:194] Registered new tserver with Master: 79ce408111cc4ace9a13544209f40d61 (127.16.99.194:44425)
I20260812 00:12:22.613665 29778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:58667
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:12:22.642161 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:35409
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:22.781090 30160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:22.781457 30160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:22.781530 30160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:22.783473 30160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:22.783759 30160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:22.783849 30160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:12:22.786254 30160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.99.212:52659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:22.786801 30160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:22.787154 30160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:22.795876 30168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:22.795876 30166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:22.796880 30165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:22.797232 30160 server_base.cc:1034] running on GCE node
I20260812 00:12:22.797490 30160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:22.797796 30160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:22.798974 30160 hybrid_clock.cc:648] HybridClock initialized: now 1786493542798950 us; error 42 us; skew 500 ppm
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:22.802520 30160 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:12:22.803073 30160 webserver.cc:460] Webserver started at http://127.16.99.195:38567/ using document root <none> and password file <none>
I20260812 00:12:22.803349 30160 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:22.803400 30160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:22.803526 30160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:22.820590 30160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "150d50788b8203953d84f9e1659f856c"
server_key_iv: "c1e34eec7cafe0d342ad0b787a86fbc2"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.821096 30160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "150d50788b8203953d84f9e1659f856c"
server_key_iv: "c1e34eec7cafe0d342ad0b787a86fbc2"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.824071 30160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260812 00:12:22.846484 30176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.850169 30160 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.001s
I20260812 00:12:22.850340 30160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "150d50788b8203953d84f9e1659f856c"
server_key_iv: "c1e34eec7cafe0d342ad0b787a86fbc2"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:22.850489 30160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:22.862550 30160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:22.865348 30160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:22.865536 30160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:22.865803 30160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:22.866218 30160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:22.866264 30160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.866297 30160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:22.866312 30160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:22.874847 30160 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:36631
I20260812 00:12:22.875018 30298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:36631 every 8 connection(s)
I20260812 00:12:22.875399 30160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:12:22 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493542, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:22.883646 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30160
I20260812 00:12:22.883754 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:22.883963 16783 external_mini_cluster.cc:1428] Setting key 3f277a52a1a829bf17aed3cb4fb5af46
I20260812 00:12:22.888468 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.876257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:56215 (local address 127.16.99.254:35409)
0812 00:12:22.876311 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:22.876314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:22.876662 (+ 348us) server_negotiation.cc:407] Connection header received
0812 00:12:22.877403 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:22.877405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:22.877419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:22.877486 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:22.878309 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.879054 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.880606 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.880887 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.880968 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.884508 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:22.884510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:22.884511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:22.884520 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:22.886445 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:22.887077 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:22.887079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:22.887079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:22.887155 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:22.887542 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:22.887543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:22.887543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:22.887714 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:22.887789 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:22.888258 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 00:12:22.888364 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:12:22.889110 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:22.876263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:35409 (local address 127.16.99.195:56215)
0812 00:12:22.876478 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:22.876537 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:22.877256 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:22.877581 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:22.877586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:22.877872 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:22.878182 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.878189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.879182 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.879183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.880486 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:22.880489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:22.881014 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:22.881015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:22.881173 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:22.881866 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:22.881879 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:22.884373 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:22.886532 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:22.886545 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:22.886548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:22.886973 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:22.887251 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:22.887252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:22.887253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:22.887452 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:22.887801 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:22.887810 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:22.888049 (+ 239us) client_negotiation.cc:769] Sending connection context
0812 00:12:22.888214 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 00:12:22.888440 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":104,"threads_started":1}
I20260812 00:12:22.889765 30299 heartbeater.cc:347] Connected to a master server at 127.16.99.254:35409
I20260812 00:12:22.889918 30299 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:22.890199 30299 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:22.890776 29778 ts_manager.cc:194] Registered new tserver with Master: a1ff5d8b1cbc4b3cbda442318ddf6924 (127.16.99.195:36631)
I20260812 00:12:22.891280 29778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:56215
I20260812 00:12:22.898797 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:22.899552 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 29864
I20260812 00:12:22.922959 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30013
I20260812 00:12:22.930928 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30160
I20260812 00:12:22.941223 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 29742
I20260812 00:12:22.950112 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:35409
--webserver_interface=127.16.99.254
--webserver_port=46483
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:35409
--ranger_config_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:23.059459 30302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:23.059695 30302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:23.059719 30302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:23.061003 30302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:23.061058 30302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:23.061074 30302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:23.061086 30302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:23.061100 30302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:23.063030 30302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:35409
--ranger_config_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=46483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:23.063369 30302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:23.063637 30302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:23.069345 30309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.069355 30308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.069355 30312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:23.069696 30302 server_base.cc:1034] running on GCE node
I20260812 00:12:23.069991 30302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:23.070323 30302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:23.071547 30302 hybrid_clock.cc:648] HybridClock initialized: now 1786493543071525 us; error 46 us; skew 500 ppm
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:23.074244 30302 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:23.074735 30302 webserver.cc:460] Webserver started at http://127.16.99.254:46483/ using document root <none> and password file <none>
I20260812 00:12:23.074999 30302 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:23.075083 30302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:23.157368 30302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:23.158232 30320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.158551 30302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:23.158686 30302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "d504710524cf47cb9ec93d9bbdf9316e"
format_stamp: "Formatted at 2026-08-12 00:12:21 on dist-test-slave-zpfg"
server_key: "43cb7fc1aaf32020e9e3caf3400e3196"
server_key_iv: "4f35995e46e71266d70eddf90da731f0"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:23.159174 30302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:23.196581 30302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:23.199074 30302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:23.199239 30302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:23.204759 30302 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:35409
I20260812 00:12:23.204790 30373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:35409 every 8 connection(s)
I20260812 00:12:23.205376 30302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:23.207077 30374 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:12:23.207828 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Bootstrap starting.
I20260812 00:12:23.210682 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30302
I20260812 00:12:23.211076 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:46159
--local_ip_for_outbound_sockets=127.16.99.193
--tserver_master_addrs=127.16.99.254:35409
--webserver_port=36979
--webserver_interface=127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:23.217729 30374 log.cc:824] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:23.219780 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:23.220161 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Bootstrap complete.
I20260812 00:12:23.223037 30374 raft_consensus.cc:348] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
I20260812 00:12:23.223344 30374 raft_consensus.cc:729] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d504710524cf47cb9ec93d9bbdf9316e, State: Initialized, Role: FOLLOWER
I20260812 00:12:23.223593 30374 consensus_queue.cc:260] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
I20260812 00:12:23.223783 30374 raft_consensus.cc:388] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:23.223845 30374 raft_consensus.cc:482] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:23.223888 30374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:12:23.225040 30374 raft_consensus.cc:504] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
I20260812 00:12:23.225198 30374 leader_election.cc:302] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d504710524cf47cb9ec93d9bbdf9316e; no voters:
I20260812 00:12:23.225435 30374 leader_election.cc:288] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:12:23.225560 30378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:12:23.225734 30374 sys_catalog.cc:564] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:23.225878 30378 raft_consensus.cc:686] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [term 2 LEADER]: Becoming Leader. State: Replica: d504710524cf47cb9ec93d9bbdf9316e, State: Running, Role: LEADER
I20260812 00:12:23.226064 30378 consensus_queue.cc:237] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } }
I20260812 00:12:23.226464 30379 sys_catalog.cc:455] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d504710524cf47cb9ec93d9bbdf9316e. Latest consensus state: current_term: 2 leader_uuid: "d504710524cf47cb9ec93d9bbdf9316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } } }
I20260812 00:12:23.226565 30379 sys_catalog.cc:458] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:23.226707 30380 sys_catalog.cc:455] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d504710524cf47cb9ec93d9bbdf9316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d504710524cf47cb9ec93d9bbdf9316e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 35409 } } }
I20260812 00:12:23.226761 30380 sys_catalog.cc:458] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:23.226861 30387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:23.227662 30387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:23.227957 30387 catalog_manager.cc:1259] Loaded cluster ID: 1e8cd6f2f5134db3b430227fd73cce20
I20260812 00:12:23.228009 30387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:23.228034 30374 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 00:12:23.232512 30387 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:23.233203 30387 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Loaded TSK: 0
I20260812 00:12:23.233547 30387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:12:23.309413 30376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:23.309670 30376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:23.309715 30376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:23.311036 30376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:23.311111 30376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:23.311164 30376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:12:23.313033 30376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:46159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=36979
--tserver_master_addrs=127.16.99.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:23.313331 30376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:23.313577 30376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:23.319252 30412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.319319 30410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.319619 30409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:23.319907 30376 server_base.cc:1034] running on GCE node
I20260812 00:12:23.320142 30376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:23.320466 30376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:23.321640 30376 hybrid_clock.cc:648] HybridClock initialized: now 1786493543321615 us; error 37 us; skew 500 ppm
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:23.325052 30376 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:12:23.325673 30376 webserver.cc:460] Webserver started at http://127.16.99.193:36979/ using document root <none> and password file <none>
I20260812 00:12:23.325963 30376 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:23.326061 30376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:23.347889 30376 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260812 00:12:23.349220 30426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.351661 30376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260812 00:12:23.351785 30376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "a6f3a05c44634b70a4ed90e457959865"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "2c1284c145e896cedd6e0513c52c31cc"
server_key_iv: "145c4befbd1be5be844bd842c94f271f"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:23.352270 30376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:23.368801 30376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:23.371538 30376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:23.371718 30376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:23.371984 30376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:23.372418 30376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:23.372462 30376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.372495 30376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:23.372514 30376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.381356 30376 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:46159
I20260812 00:12:23.381978 30376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:23.386852 30544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:46159 every 8 connection(s)
I20260812 00:12:23.389786 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30376
I20260812 00:12:23.390148 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:44425
--local_ip_for_outbound_sockets=127.16.99.194
--tserver_master_addrs=127.16.99.254:35409
--webserver_port=33643
--webserver_interface=127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:23.407501 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.383144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:35409 (local address 127.16.99.193:55651)
0812 00:12:23.387226 (+ 4082us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:23.387286 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:23.388050 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:23.388849 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:23.388854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:23.389130 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:23.389440 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.389448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.396901 (+ 7453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.396904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.398371 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.398378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.399563 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.399565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.399751 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.400551 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:23.400565 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:23.402454 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:23.405345 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:23.405356 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:23.405359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:23.405683 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:23.406026 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:23.406027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:23.406027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:23.406193 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:23.406772 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:23.406780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:23.406957 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 00:12:23.407063 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 00:12:23.407254 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4027,"thread_start_us":96,"threads_started":1}
I20260812 00:12:23.408414 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.383244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:55651 (local address 127.16.99.254:35409)
0812 00:12:23.383464 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:23.383473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:23.387475 (+ 4002us) server_negotiation.cc:407] Connection header received
0812 00:12:23.388414 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:23.388422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:23.388494 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:23.388643 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:23.389597 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.396667 (+ 7070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.398527 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.399252 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.399368 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.402725 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:23.402731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:23.402731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:23.402746 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:23.405223 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.405792 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.405796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.405797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.405926 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.406292 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.406294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.406294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.406651 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:23.406740 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:23.407875 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0812 00:12:23.408008 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260812 00:12:23.409214 30545 heartbeater.cc:347] Connected to a master server at 127.16.99.254:35409
I20260812 00:12:23.409324 30545 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.409533 30545 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
W20260812 00:12:23.410521 30545 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:35409 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 00:12:23.410607 30545 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.410681 30545 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:23.411057 30545 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.411144 30545 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
W20260812 00:12:23.411439 30545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:12:23.526275 30549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:23.526633 30549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:23.526680 30549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:23.528875 30549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:23.528951 30549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:23.528983 30549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:12:23.531226 30549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:44425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=33643
--tserver_master_addrs=127.16.99.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:23.531518 30549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:23.531813 30549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:23.539431 30555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.539431 30558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.539812 30556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:23.540154 30549 server_base.cc:1034] running on GCE node
I20260812 00:12:23.540387 30549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:23.540692 30549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:23.541891 30549 hybrid_clock.cc:648] HybridClock initialized: now 1786493543541854 us; error 50 us; skew 500 ppm
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:23.545545 30549 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:12:23.546324 30549 webserver.cc:460] Webserver started at http://127.16.99.194:33643/ using document root <none> and password file <none>
I20260812 00:12:23.546669 30549 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:23.546819 30549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:23.563289 30549 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260812 00:12:23.564740 30566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.565027 30549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:23.565145 30549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "79ce408111cc4ace9a13544209f40d61"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "0828e8a8344663ebef5c55a6cbd93479"
server_key_iv: "4b591b27756fc6476ab2a62c7e8a9a3a"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:23.565688 30549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:23.592170 30549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:23.594975 30549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:23.595170 30549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:23.595474 30549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:23.595849 30549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:23.595911 30549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.595955 30549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:23.595983 30549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.604307 30549 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:44425
I20260812 00:12:23.604498 30683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:44425 every 8 connection(s)
I20260812 00:12:23.604817 30549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:23.607127 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30549
I20260812 00:12:23.607457 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.99.212:52659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:36631
--local_ip_for_outbound_sockets=127.16.99.195
--tserver_master_addrs=127.16.99.254:35409
--webserver_port=38567
--webserver_interface=127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:23.617863 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.605675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:35409 (local address 127.16.99.194:59731)
0812 00:12:23.605845 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:23.605897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:23.606620 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:23.606996 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:23.607001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:23.607368 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:23.607966 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.607973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.608993 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.608994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.610286 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.610289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.610956 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.610958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.611124 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.612034 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:23.612046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:23.613706 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:23.615880 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:23.615889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:23.615892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:23.616280 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:23.616556 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:23.616557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:23.616557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:23.616740 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:23.617089 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:23.617097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:23.617280 (+ 183us) client_negotiation.cc:769] Sending connection context
0812 00:12:23.617405 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 00:12:23.617585 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260812 00:12:23.618461 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.605689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:59731 (local address 127.16.99.254:35409)
0812 00:12:23.605747 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:23.605749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:23.605991 (+ 242us) server_negotiation.cc:407] Connection header received
0812 00:12:23.606797 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:23.606799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:23.606816 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:23.606906 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:23.608090 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.608873 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.610438 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.610778 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.610853 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.613840 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:23.613842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:23.613843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:23.613851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:23.615767 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.616372 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.616374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.616374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.616451 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.616835 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.616836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.616836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.616990 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:23.617059 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:23.618220 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0812 00:12:23.618325 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 00:12:23.619307 30684 heartbeater.cc:347] Connected to a master server at 127.16.99.254:35409
I20260812 00:12:23.619444 30684 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.619656 30684 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
W20260812 00:12:23.620373 30684 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:35409 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 00:12:23.620465 30684 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.620543 30684 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:23.620873 30684 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.620965 30684 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
W20260812 00:12:23.621265 30684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:12:23.738828 30687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:23.739138 30687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:23.739187 30687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:23.741072 30687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:23.741144 30687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:23.741189 30687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:12:23.743619 30687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.99.212:52659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:36631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=38567
--tserver_master_addrs=127.16.99.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:23.743909 30687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:23.744249 30687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:23.751564 30697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.751597 30692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:23.751560 30695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:23.752354 30687 server_base.cc:1034] running on GCE node
I20260812 00:12:23.752604 30687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:23.752913 30687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:23.754107 30687 hybrid_clock.cc:648] HybridClock initialized: now 1786493543754091 us; error 32 us; skew 500 ppm
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:23.757783 30687 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:12:23.758585 30687 webserver.cc:460] Webserver started at http://127.16.99.195:38567/ using document root <none> and password file <none>
I20260812 00:12:23.758982 30687 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:23.759142 30687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:23.785460 30687 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260812 00:12:23.786635 30706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.786840 30687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:23.786932 30687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924"
format_stamp: "Formatted at 2026-08-12 00:12:22 on dist-test-slave-zpfg"
server_key: "150d50788b8203953d84f9e1659f856c"
server_key_iv: "c1e34eec7cafe0d342ad0b787a86fbc2"
server_key_version: "kuduclusterkey@0"
I20260812 00:12:23.787433 30687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:23.799124 30687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:23.801388 30687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:23.801553 30687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:23.801792 30687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:23.802147 30687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:23.802206 30687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.802250 30687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:23.802284 30687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:23.809190 30687 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:36631
I20260812 00:12:23.809778 30687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:23.809922 30824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:36631 every 8 connection(s)
I20260812 00:12:23.811071 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30687
Aug 12 00:12:23 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493543, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:23.822082 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.810708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:35409 (local address 127.16.99.195:33549)
0812 00:12:23.810954 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:23.811198 (+ 244us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:23.811978 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:23.812297 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:23.812302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:23.812542 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:23.812851 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.812859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.813868 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.813869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.815059 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:23.815062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.815689 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.815690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:23.815804 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.816473 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:23.816485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:23.818090 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:23.820159 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:23.820170 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:23.820173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:23.820517 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:23.820793 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:23.820794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:23.820794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:23.820954 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:23.821272 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:23.821281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:23.821471 (+ 190us) client_negotiation.cc:769] Sending connection context
0812 00:12:23.821588 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:12:23.821784 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":128,"threads_started":1}
I20260812 00:12:23.822687 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.811060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:33549 (local address 127.16.99.254:35409)
0812 00:12:23.811113 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:23.811116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:23.811415 (+ 299us) server_negotiation.cc:407] Connection header received
0812 00:12:23.812083 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:23.812085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:23.812097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:23.812160 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:23.812980 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.813749 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.815156 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.815525 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.815589 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.818216 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:23.818218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:23.818218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:23.818228 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:23.820041 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.820615 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.820617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.820617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.820715 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.821028 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.821029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.821029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.821181 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:23.821240 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:23.822493 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0812 00:12:23.822591 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":29}
I20260812 00:12:23.823470 30825 heartbeater.cc:347] Connected to a master server at 127.16.99.254:35409
I20260812 00:12:23.823576 30825 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.823791 30825 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
W20260812 00:12:23.824477 30825 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:35409 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 00:12:23.824566 30825 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.824637 30825 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:23.825136 30825 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:23.825251 30825 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
W20260812 00:12:23.825579 30825 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:12:23.827508 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:23.811709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.16.99.254:35409)
0812 00:12:23.812182 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:23.812184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:23.812191 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:23.812221 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:23.812221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:23.812233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:23.812312 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:23.812945 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.813475 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.815165 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:23.815526 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:23.815589 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:23.818220 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:23.818221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:23.818221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:23.818228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:23.826006 (+ 7778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.826577 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.826578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.826579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.826648 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:23.826925 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:23.826926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:23.826926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:23.827052 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:23.827260 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:23.827297 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 00:12:23.827385 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":453,"thread_start_us":56,"threads_started":1}
I20260812 00:12:24.411651 30545 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:24.411836 30545 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:24.412317 30337 ts_manager.cc:194] Registered new tserver with Master: a6f3a05c44634b70a4ed90e457959865 (127.16.99.193:46159)
I20260812 00:12:24.413383 30337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:55651
Aug 12 00:12:24 dist-test-slave-zpfg krb5kdc[29274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786493544, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:24.621475 30684 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:24.621641 30684 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:24.622011 30337 ts_manager.cc:194] Registered new tserver with Master: 79ce408111cc4ace9a13544209f40d61 (127.16.99.194:44425)
I20260812 00:12:24.622596 30337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:59731
I20260812 00:12:24.825793 30825 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:24.825980 30825 heartbeater.cc:511] Master 127.16.99.254:35409 requested a full tablet report, sending...
I20260812 00:12:24.826329 30337 ts_manager.cc:194] Registered new tserver with Master: a1ff5d8b1cbc4b3cbda442318ddf6924 (127.16.99.195:36631)
I20260812 00:12:24.826769 30337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:33549
I20260812 00:12:24.833467 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:24.845227 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.834517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.16.99.254:35409)
0812 00:12:24.834636 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:24.834638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:24.837424 (+ 2786us) server_negotiation.cc:407] Connection header received
0812 00:12:24.837564 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:24.837566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:24.837583 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:24.837637 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:24.838329 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.838887 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.839879 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.840205 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.840301 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.841922 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:24.841924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:24.841925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:24.841934 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:24.843862 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:24.844307 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:24.844309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:24.844309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:24.844381 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:24.844611 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:24.844612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:24.844613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:24.844743 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:24.844863 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:24.845011 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 00:12:24.845105 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260812 00:12:24.847838 30337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:24.850797 30337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 12 00:12:24 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493544, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
I20260812 00:12:24.896373 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.885413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44425 (local address 127.0.0.1:52030)
0812 00:12:24.887532 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:24.887542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:24.887585 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:24.887943 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:24.887947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:24.887949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:24.888038 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.888041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.889897 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.889899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.891346 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.891351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.896003 (+ 4652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.896004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.896126 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.896144 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 00:12:24.896214 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:12:24.896272 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2101,"thread_start_us":91,"threads_started":1}
I20260812 00:12:24.897157 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.885964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.16.99.194:44425)
0812 00:12:24.887174 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:24.887181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:24.887649 (+ 468us) server_negotiation.cc:407] Connection header received
0812 00:12:24.887707 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:24.887710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:24.887728 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:24.887807 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:24.888226 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.889797 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.891479 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.892335 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.892407 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.892476 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:24.896641 (+ 4165us) server_negotiation.cc:299] Negotiation successful
0812 00:12:24.896714 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":65,"threads_started":1}
I20260812 00:12:24.897969 30616 tablet_service.cc:1467] Processing CreateTablet for tablet d942933591ff4c41bfb66ebe982a63ff (DEFAULT_TABLE table=test-table [id=6951f81664b7438c8ff65ca1ee8672cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:24.898517 30616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d942933591ff4c41bfb66ebe982a63ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:24.898849 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.885049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46159 (local address 127.0.0.1:44570)
0812 00:12:24.887267 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:24.887331 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:24.887410 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:24.893267 (+ 5857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:24.893268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:24.893271 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:24.893333 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.893335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.894677 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.894678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.895896 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.895899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.898574 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.898575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.898656 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.898661 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:24.898725 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 00:12:24.898772 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2180,"thread_start_us":122,"threads_started":1}
I20260812 00:12:24.899186 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.885652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.16.99.193:46159)
0812 00:12:24.892976 (+ 7324us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:24.892982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:24.893004 (+ 22us) server_negotiation.cc:407] Connection header received
0812 00:12:24.893044 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:24.893047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:24.893066 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:24.893149 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:24.893414 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.894570 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.897465 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.898328 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.898444 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.898532 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:24.899050 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 00:12:24.899104 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7279,"thread_start_us":149,"threads_started":1}
I20260812 00:12:24.899887 30469 tablet_service.cc:1467] Processing CreateTablet for tablet d942933591ff4c41bfb66ebe982a63ff (DEFAULT_TABLE table=test-table [id=6951f81664b7438c8ff65ca1ee8672cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:24.900344 30469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d942933591ff4c41bfb66ebe982a63ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:24.906523 30861 tablet_bootstrap.cc:492] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: Bootstrap starting.
I20260812 00:12:24.906783 30862 tablet_bootstrap.cc:492] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865: Bootstrap starting.
I20260812 00:12:24.907354 30861 tablet_bootstrap.cc:654] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:24.907624 30862 tablet_bootstrap.cc:654] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:24.907714 30861 log.cc:824] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:24.908067 30862 log.cc:824] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:24.908823 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.886012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:36631 (local address 127.0.0.1:56750)
0812 00:12:24.897289 (+ 11277us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:24.897305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:24.897347 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:24.904994 (+ 7647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:24.904995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:24.904998 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:24.905053 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.905054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.906422 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.906423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.907496 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:24.907501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.908567 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.908569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:24.908637 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.908643 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:12:24.908677 (+ 34us) client_negotiation.cc:240] Negotiation successful
0812 00:12:24.908742 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11261,"thread_start_us":79,"threads_started":1}
I20260812 00:12:24.909261 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:24.886234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.16.99.195:36631)
0812 00:12:24.904804 (+ 18570us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:24.904809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:24.904823 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:12:24.904862 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:24.904864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:24.904882 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:24.905123 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:24.905215 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.906272 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.907616 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:24.908459 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:24.909049 (+ 590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:24.909126 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:24.909153 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 00:12:24.909200 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18523,"thread_start_us":70,"threads_started":1}
I20260812 00:12:24.910028 30755 tablet_service.cc:1467] Processing CreateTablet for tablet d942933591ff4c41bfb66ebe982a63ff (DEFAULT_TABLE table=test-table [id=6951f81664b7438c8ff65ca1ee8672cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:24.910444 30755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d942933591ff4c41bfb66ebe982a63ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:24.914840 30861 tablet_bootstrap.cc:492] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: No bootstrap required, opened a new log
I20260812 00:12:24.914933 30861 ts_tablet_manager.cc:1397] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260812 00:12:24.917747 30861 raft_consensus.cc:348] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:24.917959 30861 raft_consensus.cc:374] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:24.917994 30861 raft_consensus.cc:729] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ce408111cc4ace9a13544209f40d61, State: Initialized, Role: FOLLOWER
I20260812 00:12:24.918242 30867 tablet_bootstrap.cc:492] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Bootstrap starting.
I20260812 00:12:24.919077 30867 tablet_bootstrap.cc:654] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:24.919420 30867 log.cc:824] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:24.919739 30861 consensus_queue.cc:260] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:24.920097 30861 ts_tablet_manager.cc:1428] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 00:12:24.920271 30684 heartbeater.cc:503] Master 127.16.99.254:35409 was elected leader, sending a full tablet report...
I20260812 00:12:24.920313 30867 tablet_bootstrap.cc:492] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: No bootstrap required, opened a new log
I20260812 00:12:24.920384 30867 ts_tablet_manager.cc:1397] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:24.923151 30867 raft_consensus.cc:348] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:24.923385 30867 raft_consensus.cc:374] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:24.923435 30867 raft_consensus.cc:729] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ff5d8b1cbc4b3cbda442318ddf6924, State: Initialized, Role: FOLLOWER
I20260812 00:12:24.923594 30867 consensus_queue.cc:260] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:24.923942 30867 ts_tablet_manager.cc:1428] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260812 00:12:24.924100 30825 heartbeater.cc:503] Master 127.16.99.254:35409 was elected leader, sending a full tablet report...
I20260812 00:12:24.926229 30862 tablet_bootstrap.cc:492] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865: No bootstrap required, opened a new log
I20260812 00:12:24.926314 30862 ts_tablet_manager.cc:1397] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20260812 00:12:24.928565 30862 raft_consensus.cc:348] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:24.928711 30862 raft_consensus.cc:374] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:24.928733 30862 raft_consensus.cc:729] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f3a05c44634b70a4ed90e457959865, State: Initialized, Role: FOLLOWER
I20260812 00:12:24.928892 30862 consensus_queue.cc:260] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:24.929224 30862 ts_tablet_manager.cc:1428] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:24.929229 30545 heartbeater.cc:503] Master 127.16.99.254:35409 was elected leader, sending a full tablet report...
W20260812 00:12:25.061157 30826 tablet.cc:2369] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:25.106387 30685 tablet.cc:2369] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 00:12:25 dist-test-slave-zpfg krb5kdc[29274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493544, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for HTTP/127.16.99.212@KRBTEST.COM
W20260812 00:12:25.138463 30546 tablet.cc:2369] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:25.156345 30870 raft_consensus.cc:482] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:25.156536 30870 raft_consensus.cc:504] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:25.156980 30870 leader_election.cc:288] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6f3a05c44634b70a4ed90e457959865 (127.16.99.193:46159), 79ce408111cc4ace9a13544209f40d61 (127.16.99.194:44425)
I20260812 00:12:25.161556 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.157217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:38325 (local address 127.16.99.193:46159)
0812 00:12:25.157554 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:25.157557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:25.157566 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:25.157599 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:25.157600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:25.157611 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:25.157662 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:25.158078 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.158890 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.160531 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.161111 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.161165 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.161175 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:25.161434 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 00:12:25.161479 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260812 00:12:25.161556 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.157819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:42369 (local address 127.16.99.194:44425)
0812 00:12:25.157866 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:25.157868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:25.157875 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:25.157900 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:25.157901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:25.157910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:25.157963 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:25.158146 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.158890 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.160544 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.161111 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.161165 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.161175 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:25.161434 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 00:12:25.161479 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:12:25.161887 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.157414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44425 (local address 127.16.99.195:42369)
0812 00:12:25.157547 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:25.157557 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:25.157598 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:25.158007 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:25.158008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:25.158010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:25.158066 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.158068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.158999 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.159000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.160450 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.160452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.161263 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.161264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.161341 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.161347 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:12:25.161730 (+ 383us) client_negotiation.cc:240] Negotiation successful
0812 00:12:25.161791 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":67,"thread_start_us":83,"threads_started":1}
I20260812 00:12:25.162264 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.157153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46159 (local address 127.16.99.195:38325)
0812 00:12:25.157355 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:25.157367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:25.157423 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:25.157861 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:25.157862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:25.157865 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:25.157986 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.157989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.158999 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.159000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.160416 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:25.160420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.161257 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.161258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:25.161341 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.161348 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 00:12:25.162120 (+ 772us) client_negotiation.cc:240] Negotiation successful
0812 00:12:25.162181 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":74,"thread_start_us":139,"threads_started":1}
I20260812 00:12:25.162254 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d942933591ff4c41bfb66ebe982a63ff" candidate_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ce408111cc4ace9a13544209f40d61" is_pre_election: true
I20260812 00:12:25.162465 30637 raft_consensus.cc:2454] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ff5d8b1cbc4b3cbda442318ddf6924 in term 0.
I20260812 00:12:25.162742 30709 leader_election.cc:302] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ce408111cc4ace9a13544209f40d61, a1ff5d8b1cbc4b3cbda442318ddf6924; no voters:
I20260812 00:12:25.162953 30870 raft_consensus.cc:2781] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:25.163017 30870 raft_consensus.cc:482] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:25.163043 30870 raft_consensus.cc:3037] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:25.163146 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d942933591ff4c41bfb66ebe982a63ff" candidate_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f3a05c44634b70a4ed90e457959865" is_pre_election: true
I20260812 00:12:25.163300 30487 raft_consensus.cc:2454] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ff5d8b1cbc4b3cbda442318ddf6924 in term 0.
I20260812 00:12:25.164080 30870 raft_consensus.cc:504] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:25.164409 30870 leader_election.cc:288] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [CANDIDATE]: Term 1 election: Requested vote from peers a6f3a05c44634b70a4ed90e457959865 (127.16.99.193:46159), 79ce408111cc4ace9a13544209f40d61 (127.16.99.194:44425)
I20260812 00:12:25.164932 30487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d942933591ff4c41bfb66ebe982a63ff" candidate_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6f3a05c44634b70a4ed90e457959865"
I20260812 00:12:25.164918 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d942933591ff4c41bfb66ebe982a63ff" candidate_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ce408111cc4ace9a13544209f40d61"
I20260812 00:12:25.165011 30637 raft_consensus.cc:3037] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:25.165011 30487 raft_consensus.cc:3037] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:25.166031 30487 raft_consensus.cc:2454] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ff5d8b1cbc4b3cbda442318ddf6924 in term 1.
I20260812 00:12:25.166071 30637 raft_consensus.cc:2454] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ff5d8b1cbc4b3cbda442318ddf6924 in term 1.
I20260812 00:12:25.166330 30709 leader_election.cc:302] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ce408111cc4ace9a13544209f40d61, a1ff5d8b1cbc4b3cbda442318ddf6924; no voters:
I20260812 00:12:25.166533 30870 raft_consensus.cc:2781] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:25.166739 30870 raft_consensus.cc:686] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 1 LEADER]: Becoming Leader. State: Replica: a1ff5d8b1cbc4b3cbda442318ddf6924, State: Running, Role: LEADER
I20260812 00:12:25.166860 30870 consensus_queue.cc:237] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } }
I20260812 00:12:25.167742 30337 catalog_manager.cc:5495] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1ff5d8b1cbc4b3cbda442318ddf6924 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1ff5d8b1cbc4b3cbda442318ddf6924" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 36631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 } health_report { overall_health: UNKNOWN } } }
I20260812 00:12:25.175684 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.171660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.16.99.195:36631)
0812 00:12:25.171720 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:25.171723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:25.171765 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:12:25.171852 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:25.171853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:25.171879 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:25.171946 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:25.172137 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.172909 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.174384 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:25.174717 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:25.174789 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:25.175019 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:25.175200 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:25.175285 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:25.175572 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 00:12:25.175598 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:12:25.178056 30637 raft_consensus.cc:1261] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 1 FOLLOWER]: Refusing update from remote peer a1ff5d8b1cbc4b3cbda442318ddf6924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:25.178460 30487 raft_consensus.cc:1261] T d942933591ff4c41bfb66ebe982a63ff P a6f3a05c44634b70a4ed90e457959865 [term 1 FOLLOWER]: Refusing update from remote peer a1ff5d8b1cbc4b3cbda442318ddf6924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:25.179041 30875 consensus_queue.cc:1035] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ce408111cc4ace9a13544209f40d61" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:25.179195 30870 consensus_queue.cc:1035] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6f3a05c44634b70a4ed90e457959865" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:25.187000 30879 mvcc.cc:205] Tried to move back new op lower bound from 7317477561044918272 to 7317477561004007424. Current Snapshot: MvccSnapshot[applied={T|T < 7317477561044918272}]
I20260812 00:12:25.187834 30882 mvcc.cc:205] Tried to move back new op lower bound from 7317477561044918272 to 7317477561004007424. Current Snapshot: MvccSnapshot[applied={T|T < 7317477561044918272}]
I20260812 00:12:25.188371 30880 mvcc.cc:205] Tried to move back new op lower bound from 7317477561044918272 to 7317477561004007424. Current Snapshot: MvccSnapshot[applied={T|T < 7317477561044918272}]
I20260812 00:12:25.192373 30336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:25.192529 30336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:25.193529 30336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d504710524cf47cb9ec93d9bbdf9316e: Sending DeleteTablet for 3 replicas of tablet d942933591ff4c41bfb66ebe982a63ff
I20260812 00:12:25.194002 30469 tablet_service.cc:1514] Processing DeleteTablet for tablet d942933591ff4c41bfb66ebe982a63ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:25 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:44570
I20260812 00:12:25.194037 30616 tablet_service.cc:1514] Processing DeleteTablet for tablet d942933591ff4c41bfb66ebe982a63ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:25 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:52030
I20260812 00:12:25.194465 30755 tablet_service.cc:1514] Processing DeleteTablet for tablet d942933591ff4c41bfb66ebe982a63ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:25 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:56750
I20260812 00:12:25.194764 30895 tablet_replica.cc:330] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: stopping tablet replica
I20260812 00:12:25.194931 30895 raft_consensus.cc:2229] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:25.195142 30895 raft_consensus.cc:2258] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:25.195156 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30376
I20260812 00:12:25.196871 30895 ts_tablet_manager.cc:1905] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:25.198989 30895 ts_tablet_manager.cc:1918] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:25.199198 30895 log.cc:1196] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/d942933591ff4c41bfb66ebe982a63ff
I20260812 00:12:25.199625 30895 ts_tablet_manager.cc:1939] T d942933591ff4c41bfb66ebe982a63ff P a1ff5d8b1cbc4b3cbda442318ddf6924: Deleting consensus metadata
I20260812 00:12:25.200376 30324 catalog_manager.cc:4841] TS a1ff5d8b1cbc4b3cbda442318ddf6924 (127.16.99.195:36631): tablet d942933591ff4c41bfb66ebe982a63ff (table test-table [id=6951f81664b7438c8ff65ca1ee8672cf]) successfully deleted
W20260812 00:12:25.212734 30323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:46159 (error 108)
I20260812 00:12:25.213079 30893 tablet_replica.cc:330] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: stopping tablet replica
I20260812 00:12:25.213326 30893 raft_consensus.cc:2229] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:25.213555 30893 raft_consensus.cc:2258] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:25.214557 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.214417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46159 (local address 127.0.0.1:44578)
0812 00:12:25.214474 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:25.214519 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 00:12:25.214695 30323 catalog_manager.cc:4584] TS a6f3a05c44634b70a4ed90e457959865 (127.16.99.193:46159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d942933591ff4c41bfb66ebe982a63ff: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46159: connect: Connection refused (error 111)
I20260812 00:12:25.214910 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30549
I20260812 00:12:25.215337 30893 ts_tablet_manager.cc:1905] T d942933591ff4c41bfb66ebe982a63ff P 79ce408111cc4ace9a13544209f40d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:25.248999 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30687
I20260812 00:12:25.253047 30855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:25.252833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44425 (local address 127.0.0.1:52040)
0812 00:12:25.252948 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:25.252991 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.194:44425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 00:12:25.261166 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30302
I20260812 00:12:25.274474 16783 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 00:12:25.630089 16783 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 00:12:25.630261 16783 mini_ranger.cc:66] Stopping Ranger...
I20260812 00:12:25.985698 16783 mini_ranger.cc:68] Stopped Ranger
2026-08-12 00:12:25.985 UTC [29305] LOG: received smart shutdown request
2026-08-12 00:12:25.990 UTC [29305] LOG: background worker "logical replication launcher" (PID 29316) exited with exit code 1
2026-08-12 00:12:25.991 UTC [29309] LOG: shutting down
2026-08-12 00:12:26.213 UTC [29305] LOG: database system is shut down
2026-08-12T00:12:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53661 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:26 dist-test-slave-zpfg krb5kdc[30906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:26 dist-test-slave-zpfg krb5kdc[30906](info): set up 2 sockets
Aug 12 00:12:26 dist-test-slave-zpfg krb5kdc[30906](info): commencing operation
krb5kdc: starting...
W20260812 00:12:27.468499 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:12:27 dist-test-slave-zpfg krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:12:27.606554 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:46183
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:46183,127.16.99.253:34975,127.16.99.252:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:27.701406 30919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:27.701785 30919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:27.701838 30919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:27.703748 30919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:27.703853 30919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:27.703881 30919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:27.703905 30919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:27.703941 30919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:27.706028 30919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:46183,127.16.99.253:34975,127.16.99.252:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:46183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:27.706449 30919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:27.706745 30919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:27.712949 30924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:27.712941 30928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:27.712941 30925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:27.713045 30919 server_base.cc:1034] running on GCE node
I20260812 00:12:27.713555 30919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:27.713912 30919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:27.715165 30919 hybrid_clock.cc:648] HybridClock initialized: now 1786493547715064 us; error 40 us; skew 500 ppm
Aug 12 00:12:27 dist-test-slave-zpfg krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:27.718160 30919 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:27.718870 30919 webserver.cc:460] Webserver started at http://127.16.99.254:35879/ using document root <none> and password file <none>
I20260812 00:12:27.719156 30919 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:27.719218 30919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:27.719417 30919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:27.720839 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "a2bfb53b6e5a4c698a7625da547f5c1f"
format_stamp: "Formatted at 2026-08-12 00:12:27 on dist-test-slave-zpfg"
server_key: "220b06cc90c2f812c1ad66db3cac5b6c"
server_key_iv: "c472cc25d1bf2c715cbb71cbaea22cde"
server_key_version: "encryptionkey@0"
I20260812 00:12:27.721269 30919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "a2bfb53b6e5a4c698a7625da547f5c1f"
format_stamp: "Formatted at 2026-08-12 00:12:27 on dist-test-slave-zpfg"
server_key: "220b06cc90c2f812c1ad66db3cac5b6c"
server_key_iv: "c472cc25d1bf2c715cbb71cbaea22cde"
server_key_version: "encryptionkey@0"
I20260812 00:12:27.723043 30919 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260812 00:12:27.724054 30935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:27.724309 30919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:27.724442 30919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "a2bfb53b6e5a4c698a7625da547f5c1f"
format_stamp: "Formatted at 2026-08-12 00:12:27 on dist-test-slave-zpfg"
server_key: "220b06cc90c2f812c1ad66db3cac5b6c"
server_key_iv: "c472cc25d1bf2c715cbb71cbaea22cde"
server_key_version: "encryptionkey@0"
I20260812 00:12:27.724537 30919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:27.754010 30919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:27.756881 30919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:27.757121 30919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:27.762353 30919 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:46183
I20260812 00:12:27.762435 30990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:46183 every 8 connection(s)
I20260812 00:12:27.762878 30919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:27.763733 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30919
I20260812 00:12:27.763877 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:12:27.764011 30991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:27.764072 16783 external_mini_cluster.cc:1428] Setting key 08212ce6bae8d238eb874cf116867146
I20260812 00:12:27.765763 30991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:12:27 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.253@KRBTEST.COM" created.
I20260812 00:12:27.778424 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.766677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.16.99.254:46183)
0812 00:12:27.766900 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:27.766907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:27.766964 (+ 57us) server_negotiation.cc:407] Connection header received
0812 00:12:27.767595 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:27.767597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:27.767617 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:27.767752 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:27.768610 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.769616 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.771132 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.771509 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.771609 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.774540 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:27.774550 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:27.774550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:27.774559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:27.776385 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:27.776895 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:27.776899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:27.776899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:27.776990 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:27.777292 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:27.777293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:27.777294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:27.777597 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:27.777696 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:27.777889 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 00:12:27.778078 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260812 00:12:27.778434 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.766648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:46183 (local address 127.0.0.1:36030)
0812 00:12:27.766832 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.766878 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:27.767428 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:27.767795 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:27.767798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:27.768209 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:27.768470 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.768477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.769734 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.769735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.771035 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.771038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.771608 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.771610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.771708 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.772376 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:27.772388 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:27.774425 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:27.776521 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.776529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.776531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.776809 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:27.777059 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.777060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.777060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.777190 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:27.777672 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:27.777677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:27.777748 (+ 71us) client_negotiation.cc:769] Sending connection context
0812 00:12:27.777862 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 00:12:27.778127 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":118,"thread_start_us":89,"threads_started":1}
I20260812 00:12:27.779644 30991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:12:27.780090 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.779962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49276)
0812 00:12:27.780016 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.780065 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 00:12:27.780283 30940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
I20260812 00:12:27.780889 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.780734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49286)
0812 00:12:27.780818 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.780863 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 00:12:27.781039 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:34975: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab.
Entry for principal kudu/127.16.99.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab.
I20260812 00:12:27.799146 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:34975
--webserver_interface=127.16.99.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab
--principal=kudu/127.16.99.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:46183,127.16.99.253:34975,127.16.99.252:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:27.830408 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } attempt: 1
I20260812 00:12:27.831032 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.830876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49294)
0812 00:12:27.830944 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.830998 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 00:12:27.831521 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.831418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49302)
0812 00:12:27.831465 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.831497 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:12:27.831636 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:34975: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
W20260812 00:12:27.889974 30998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:27.890246 30998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:27.890277 30998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:27.891795 30998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:27.891876 30998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:27.891894 30998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:27.891909 30998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:27.891925 30998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:27.894078 30998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:46183,127.16.99.253:34975,127.16.99.252:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab
--principal=kudu/127.16.99.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:34975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.99.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:27.894488 30998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:27.894927 30998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:12:27.899775 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } attempt: 2
I20260812 00:12:27.900432 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.900131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49312)
0812 00:12:27.900224 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.900383 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 00:12:27.901006 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.900866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49324)
0812 00:12:27.900940 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.900978 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 00:12:27.901262 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:34975: Network error: Client connection negotiation failed: client connection to 127.16.99.253:34975: connect: Connection refused (error 111)
W20260812 00:12:27.901546 31004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:27.901546 31005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:27.901805 31007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:27.901932 30998 server_base.cc:1034] running on GCE node
I20260812 00:12:27.902164 30998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:27.902603 30998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:27.903806 30998 hybrid_clock.cc:648] HybridClock initialized: now 1786493547903776 us; error 68 us; skew 500 ppm
Aug 12 00:12:27 dist-test-slave-zpfg krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:27.907315 30998 init.cc:377] Logged in from keytab as kudu/127.16.99.253@KRBTEST.COM (short username kudu)
I20260812 00:12:27.907940 30998 webserver.cc:460] Webserver started at http://127.16.99.253:42265/ using document root <none> and password file <none>
I20260812 00:12:27.908252 30998 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:27.908340 30998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:27.908526 30998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:27.909878 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data/instance:
uuid: "23c0ed05a4f4451198cdce1a2e9b0f31"
format_stamp: "Formatted at 2026-08-12 00:12:27 on dist-test-slave-zpfg"
server_key: "3c2a446c852218d4dbb881d5838e2ff4"
server_key_iv: "0c4bcb869fadf36eda2f0570e00c0cd3"
server_key_version: "encryptionkey@0"
I20260812 00:12:27.910316 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/wal/instance:
uuid: "23c0ed05a4f4451198cdce1a2e9b0f31"
format_stamp: "Formatted at 2026-08-12 00:12:27 on dist-test-slave-zpfg"
server_key: "3c2a446c852218d4dbb881d5838e2ff4"
server_key_iv: "0c4bcb869fadf36eda2f0570e00c0cd3"
server_key_version: "encryptionkey@0"
I20260812 00:12:27.912122 30998 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:27.913268 31015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:27.913537 30998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:27.913651 30998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/wal
uuid: "23c0ed05a4f4451198cdce1a2e9b0f31"
format_stamp: "Formatted at 2026-08-12 00:12:27 on dist-test-slave-zpfg"
server_key: "3c2a446c852218d4dbb881d5838e2ff4"
server_key_iv: "0c4bcb869fadf36eda2f0570e00c0cd3"
server_key_version: "encryptionkey@0"
I20260812 00:12:27.913723 30998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:27.933579 30998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:27.936486 30998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:27.936635 30998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:27.941833 30998 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.253:34975
I20260812 00:12:27.941874 31071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.253:34975 every 8 connection(s)
I20260812 00:12:27.942327 30998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
I20260812 00:12:27.943567 31072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:27.945518 31072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:12:27.946134 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 30998
I20260812 00:12:27.946254 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-1/wal/instance
I20260812 00:12:27.946563 16783 external_mini_cluster.cc:1428] Setting key 16006e46af0832fef192abffa9a405de
Aug 12 00:12:27 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.252@KRBTEST.COM; defaulting to no policy
I20260812 00:12:27.957863 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.946663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.16.99.254:46183)
0812 00:12:27.946747 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:27.946750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:27.946968 (+ 218us) server_negotiation.cc:407] Connection header received
0812 00:12:27.947425 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:27.947427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:27.947446 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:27.947537 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:27.948266 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.948960 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.950580 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.950990 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.951082 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.953842 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:27.953844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:27.953845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:27.953854 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:27.956155 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:27.956623 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:27.956624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:27.956625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:27.956705 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:27.957178 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:27.957179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:27.957179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:27.957310 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:27.957385 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:27.957707 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 00:12:27.957791 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:12:27.958205 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.946601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:46183 (local address 127.0.0.1:36038)
0812 00:12:27.946818 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.946863 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:27.947326 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:27.947616 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:27.947620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:27.947940 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:27.948161 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.948165 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.949243 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.949244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.950477 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.950480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.951245 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.951246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.951373 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.952184 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:27.952194 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:27.953712 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:27.956280 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.956288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.956290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.956530 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:27.956828 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.956829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.956829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.957010 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:27.957399 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:27.957407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:27.957562 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 00:12:27.957664 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 00:12:27.957889 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":115,"threads_started":1}
I20260812 00:12:27.958971 31072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.252@KRBTEST.COM" created.
Aug 12 00:12:27 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:12:27.969004 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.959459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49332)
0812 00:12:27.959497 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.959509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:27.959559 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:27.959823 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:27.959824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:27.959991 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:27.960045 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.960047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.961122 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.961123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.961723 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.961725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.962513 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.962515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.962582 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.963072 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:27.963078 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:27.964637 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:27.967677 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.967679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.967679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.967872 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:27.968169 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.968170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.968170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.968237 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:27.968686 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:27.968687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:27.968737 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 00:12:27.968820 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:12:27.968915 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 00:12:27.969177 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.959431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.16.99.253:34975)
0812 00:12:27.959658 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:27.959663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:27.959690 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:12:27.959717 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:27.959719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:27.959731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:27.959822 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:27.960139 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.961011 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.961844 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.962417 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.962507 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.964736 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:27.964748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:27.964748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:27.964757 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:27.967537 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:27.967988 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:27.967993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:27.967993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:27.968069 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:27.968324 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:27.968325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:27.968325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:27.968567 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:27.968678 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:27.968896 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 00:12:27.968999 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":134,"threads_started":1}
I20260812 00:12:27.969939 31072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:12:27.970407 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.970212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40706)
0812 00:12:27.970273 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.970325 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 00:12:27.970584 31016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
I20260812 00:12:27.971215 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.970983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40718)
0812 00:12:27.971062 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.971191 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 00:12:27.971439 31072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:34679: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab.
Entry for principal kudu/127.16.99.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab.
I20260812 00:12:27.982900 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:34679
--webserver_interface=127.16.99.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab
--principal=kudu/127.16.99.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:46183,127.16.99.253:34975,127.16.99.252:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:27.992451 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } attempt: 3
Aug 12 00:12:27 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:12:28.000614 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.992890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49342)
0812 00:12:27.992960 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:27.992974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:27.993058 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:27.993232 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:27.993233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:27.993429 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:27.993502 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.993505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.994401 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.994403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.994934 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:27.994936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.995350 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.995351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:27.995441 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.996021 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:27.996026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:27.997634 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:27.999501 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.999502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.999503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.999693 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:27.999933 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:27.999934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:27.999934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:27.999989 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.000339 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:28.000340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:28.000389 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 00:12:28.000449 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:12:28.000536 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 00:12:28.000684 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:27.992968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.16.99.253:34975)
0812 00:12:27.993044 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:27.993046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:27.993059 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:12:27.993142 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:27.993143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:27.993155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:27.993225 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:27.993670 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.994247 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.995028 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:27.995263 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:27.995350 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:27.997741 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:27.997743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:27.997743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:27.997752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:27.999402 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:27.999787 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:27.999789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:27.999789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:27.999859 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.000060 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.000061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.000061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.000254 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.000330 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.000495 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.000586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 00:12:28.001241 30991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:12:28.001618 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.001498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40728)
0812 00:12:28.001556 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.001592 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 00:12:28.001959 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.001871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40730)
0812 00:12:28.001918 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.001943 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260812 00:12:28.002071 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:34679: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
I20260812 00:12:28.020722 31072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } attempt: 1
I20260812 00:12:28.021274 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.021124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40742)
0812 00:12:28.021199 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.021241 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 00:12:28.021708 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.021607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40748)
0812 00:12:28.021656 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.021687 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:12:28.021868 31072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:34679: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
I20260812 00:12:28.033247 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } attempt: 1
I20260812 00:12:28.033881 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.033729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40760)
0812 00:12:28.033804 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.033849 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 00:12:28.034404 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.034200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40774)
0812 00:12:28.034258 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.034319 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 00:12:28.034520 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:34679: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
W20260812 00:12:28.074548 31079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:28.074844 31079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:28.074891 31079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:28.076246 31079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:28.076332 31079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:28.076362 31079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:28.076395 31079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:28.076428 31079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:28.078258 31079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:46183,127.16.99.253:34975,127.16.99.252:34679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab
--principal=kudu/127.16.99.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.99.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:28.078636 31079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:28.078926 31079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:12:28.084167 31079 server_base.cc:1034] running on GCE node
W20260812 00:12:28.084259 31086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:28.084120 31088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:28.084124 31085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:28.084694 31079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:28.084995 31079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:28.086175 31079 hybrid_clock.cc:648] HybridClock initialized: now 1786493548086149 us; error 43 us; skew 500 ppm
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:28.089214 31079 init.cc:377] Logged in from keytab as kudu/127.16.99.252@KRBTEST.COM (short username kudu)
I20260812 00:12:28.089756 31079 webserver.cc:460] Webserver started at http://127.16.99.252:39233/ using document root <none> and password file <none>
I20260812 00:12:28.090047 31079 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:28.090121 31079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:28.090042 31072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } attempt: 2
I20260812 00:12:28.090298 31079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:28.090607 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.090463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40788)
0812 00:12:28.090543 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.090580 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 00:12:28.090946 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.090882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40802)
0812 00:12:28.090906 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.090931 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260812 00:12:28.091060 31072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:34679: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
I20260812 00:12:28.091699 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data/instance:
uuid: "71c27c3822c94478bd360eed55944f01"
format_stamp: "Formatted at 2026-08-12 00:12:28 on dist-test-slave-zpfg"
server_key: "899436ac4b2cf901978eea359c26dbcd"
server_key_iv: "44b2dfec2f77afe96af8e536d7cc05dd"
server_key_version: "encryptionkey@0"
I20260812 00:12:28.092094 31079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/wal/instance:
uuid: "71c27c3822c94478bd360eed55944f01"
format_stamp: "Formatted at 2026-08-12 00:12:28 on dist-test-slave-zpfg"
server_key: "899436ac4b2cf901978eea359c26dbcd"
server_key_iv: "44b2dfec2f77afe96af8e536d7cc05dd"
server_key_version: "encryptionkey@0"
I20260812 00:12:28.093565 31079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:12:28.094522 31097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:28.094745 31079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:28.094885 31079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/wal
uuid: "71c27c3822c94478bd360eed55944f01"
format_stamp: "Formatted at 2026-08-12 00:12:28 on dist-test-slave-zpfg"
server_key: "899436ac4b2cf901978eea359c26dbcd"
server_key_iv: "44b2dfec2f77afe96af8e536d7cc05dd"
server_key_version: "encryptionkey@0"
I20260812 00:12:28.094982 31079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:28.109781 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } attempt: 2
I20260812 00:12:28.110291 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.110150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40816)
0812 00:12:28.110205 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.110241 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 00:12:28.110754 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.110648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40830)
0812 00:12:28.110699 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.110726 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:12:28.110884 30991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:34679: Network error: Client connection negotiation failed: client connection to 127.16.99.252:34679: connect: Connection refused (error 111)
I20260812 00:12:28.111572 31079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:28.113952 31079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:28.114090 31079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:28.118916 31079 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.252:34679
I20260812 00:12:28.118950 31151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.252:34679 every 8 connection(s)
I20260812 00:12:28.119414 31079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
I20260812 00:12:28.119621 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 31079
I20260812 00:12:28.119768 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786493381608766-16783-0/minicluster-data/master-2/wal/instance
I20260812 00:12:28.120021 16783 external_mini_cluster.cc:1428] Setting key a3be1c866106d32bbda4c01fb60cf1e7
I20260812 00:12:28.120596 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:28.123138 31152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:28.131791 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.120632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.16.99.252:34679)
0812 00:12:28.120856 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.120867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.120909 (+ 42us) server_negotiation.cc:407] Connection header received
0812 00:12:28.121405 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.121414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.121468 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.121651 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.122422 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.123431 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.124414 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.124995 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.125129 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.127802 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.127810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.127810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.127817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.129857 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.130315 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.130320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.130320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.130463 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.130708 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.130709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.130709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.131079 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.131256 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.131335 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.131473 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260812 00:12:28.133616 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.124269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:46183 (local address 127.0.0.1:36050)
0812 00:12:28.124464 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.124505 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.124588 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.124732 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.124734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.125001 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.125083 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.125085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.125983 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.125984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.126661 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.126664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.127079 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.127081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.127160 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.127797 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.127807 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.129568 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:28.131552 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.131557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.131560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.131802 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.132220 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.132221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.132221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.132292 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.133011 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:28.133029 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:28.133189 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 00:12:28.133318 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 00:12:28.133484 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":139,"threads_started":1}
I20260812 00:12:28.133625 30995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.124190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.16.99.254:46183)
0812 00:12:28.124269 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.124272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.124593 (+ 321us) server_negotiation.cc:407] Connection header received
0812 00:12:28.124631 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.124632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.124646 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.124718 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.125179 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.125875 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.126754 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.126970 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.127063 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.129666 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.129667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.129668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.129677 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.131460 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.132064 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.132066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.132067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.132138 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.132525 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.132526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.132527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.132708 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.132798 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.133400 (+ 602us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.133500 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:12:28.134454 31152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:12:28.142205 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.133016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.16.99.254:46183)
0812 00:12:28.133238 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.133240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.133250 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:28.133273 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.133274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.133287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.133366 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.134056 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.134880 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.135993 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.136270 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.136435 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.139144 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.139145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.139146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.139152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.140931 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.141385 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.141386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.141386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.141442 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.141657 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.141657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.141658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.141770 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.141901 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.142002 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.142068 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":117,"threads_started":1}
I20260812 00:12:28.143384 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.134803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:34975 (local address 127.0.0.1:49356)
0812 00:12:28.134862 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.134877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.134928 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.135071 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.135072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.135222 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.135261 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.135262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.136205 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.136206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.136804 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.136807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.137182 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.137183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.137243 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.137675 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.137679 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.139515 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:28.141922 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.141923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.141924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.142149 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.142608 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.142609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.142610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.142681 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.143045 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:28.143046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:28.143099 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 00:12:28.143169 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:12:28.143284 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:12:28.143395 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.134868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.16.99.253:34975)
0812 00:12:28.134916 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.134918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.134929 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:28.134987 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.134988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.135000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.135069 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.135350 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.136123 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.136906 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.137108 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.137185 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.139608 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.139610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.139610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.139617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.141826 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.142271 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.142273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.142273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.142364 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.142767 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.142768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.142768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.142945 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.143057 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.143221 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.143311 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:28.143728 31152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493548, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:12:28.151844 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.144032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40840)
0812 00:12:28.144077 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.144091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.144138 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.144246 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.144247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.144404 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.144441 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.144442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.145491 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.145492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.146119 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.146121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.146627 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.146628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.146691 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.147149 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.147160 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.148855 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:28.150609 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.150611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.150611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.150840 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.151055 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.151056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.151056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.151101 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.151420 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:28.151421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:28.151482 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 00:12:28.151566 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 00:12:28.151694 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260812 00:12:28.151914 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.143011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.16.99.253:34975)
0812 00:12:28.143206 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.143208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.143215 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:28.143234 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.143235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.143249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.143311 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.144030 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.144751 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.145448 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.145673 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.145753 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.148569 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.148571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.148571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.148580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.150219 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.150642 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.150643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.150643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.150696 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.151081 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.151082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.151082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.151266 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.151463 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.151668 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.151785 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":139,"threads_started":1}
I20260812 00:12:28.151921 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.144065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.16.99.252:34679)
0812 00:12:28.144098 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.144100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.144124 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:12:28.144187 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.144188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.144195 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.144233 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.144707 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.145350 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.146215 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.146545 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.146629 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.148962 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.148964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.148964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.148972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.150526 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.150936 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.150937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.150938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.150991 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.151185 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.151186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.151186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.151314 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.151377 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.151610 (+ 233us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.151784 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":15}
I20260812 00:12:28.153888 31152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01: Bootstrap starting.
I20260812 00:12:28.154590 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:12:28.154824 31152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:28.155126 31152 log.cc:824] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:28.156339 31152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01: No bootstrap required, opened a new log
I20260812 00:12:28.158900 31152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.159087 31152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:28.159138 31152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c27c3822c94478bd360eed55944f01, State: Initialized, Role: FOLLOWER
I20260812 00:12:28.159279 31152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.159626 31152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:28.160754 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.161000 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:12:28.162024 31178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:28.162147 31178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:28.162088 31079 master_runner.cc:186] Error getting master registration for 127.16.99.254:46183: OK, instance_id { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" instance_seqno: 1786493547760797 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:12:28.166045 31079 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:12:28.167255 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.156410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.16.99.252:34679)
0812 00:12:28.156451 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.156454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.156610 (+ 156us) server_negotiation.cc:407] Connection header received
0812 00:12:28.156783 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.156784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.156797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.156840 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.157666 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.158510 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.159676 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.160002 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.160045 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.163917 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.163919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.163919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.163927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.165666 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.166150 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.166151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.166152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.166217 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.166509 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.166510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.166510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.166691 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.166788 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.167047 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.167145 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:28.167618 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.156615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.16.99.253:34975)
0812 00:12:28.156656 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.156658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.156933 (+ 275us) server_negotiation.cc:407] Connection header received
0812 00:12:28.157055 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.157056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.157069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.157102 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.157855 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.158817 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.159820 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.160155 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.160197 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.162801 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.162803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.162803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.162812 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.165858 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.166333 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.166334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.166335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.166436 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.166700 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.166701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.166702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.166884 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.166981 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.167391 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.167491 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:28.168799 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.156293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.16.99.254:46183)
0812 00:12:28.156351 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.156354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.156480 (+ 126us) server_negotiation.cc:407] Connection header received
0812 00:12:28.156541 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.156542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.156557 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.156626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.157496 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.158375 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.159859 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.160265 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.160322 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.162223 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.162225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.162225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.162235 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.167369 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.167801 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.167802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.167803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.167876 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.168159 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.168160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.168161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.168314 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.168436 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.168597 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.168688 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 00:12:28.182231 31072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } attempt: 3
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:12:28.191381 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.182774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.16.99.252:34679)
0812 00:12:28.182872 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.182874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.182890 (+ 16us) server_negotiation.cc:407] Connection header received
0812 00:12:28.182914 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.182915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.182923 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.182973 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.183538 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.184041 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.185157 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.185352 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.185412 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.187677 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.187678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.187679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.187687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.189896 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.190415 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.190417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.190417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.190487 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.190790 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.190791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.190792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.190943 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.191016 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.191230 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.191308 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 00:12:28.191426 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.182674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40864)
0812 00:12:28.182756 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.182774 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.182843 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.183005 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.183006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.183253 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.183319 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.183322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.184139 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.184140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.185061 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.185064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.185439 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.185441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.185509 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.186049 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.186055 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.187579 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:28.189997 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.189999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.189999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.190262 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.190568 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.190569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.190569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.190658 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.191041 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:28.191042 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:28.191102 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 00:12:28.191185 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:12:28.191313 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 00:12:28.193343 31072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: Bootstrap starting.
I20260812 00:12:28.194095 31072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:28.194523 31072 log.cc:824] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:28.195395 31072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: No bootstrap required, opened a new log
I20260812 00:12:28.196959 31072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.197130 31072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:28.197268 31072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c0ed05a4f4451198cdce1a2e9b0f31, State: Initialized, Role: FOLLOWER
I20260812 00:12:28.197423 31072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.197751 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.197836 31072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:28.197873 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:12:28.200441 31193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:28.200567 31193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:28.200697 30998 master_runner.cc:186] Error getting master registration for 127.16.99.254:46183: OK, instance_id { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" instance_seqno: 1786493547760797 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:12:28.203253 30998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:12:28.210140 30991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } attempt: 3
Aug 12 00:12:28 dist-test-slave-zpfg krb5kdc[30906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493547, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:12:28.220176 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.210749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.16.99.252:34679)
0812 00:12:28.210801 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:28.210803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:28.210813 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:28.210843 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:28.210844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:28.210855 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:28.210938 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:28.211361 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.212219 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.213607 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.213919 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.214007 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.216602 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:28.216604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:28.216605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:28.216614 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:28.218670 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.219133 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.219135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.219135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.219208 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:28.219463 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:28.219464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:28.219465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:28.219628 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:28.219789 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:28.219974 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 00:12:28.220069 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:12:28.220168 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:28.210640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:34679 (local address 127.0.0.1:40870)
0812 00:12:28.210703 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:28.210715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:28.210757 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:28.210962 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:28.210964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:28.211193 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:28.211258 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.211260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.212336 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.212338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.213221 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:28.213224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:28.214027 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:28.214028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:28.214109 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:28.214720 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:28.214727 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:28.216507 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:28.218780 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.218781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.218782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.219037 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.219294 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:28.219295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:28.219296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:28.219376 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:28.219787 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:28.219788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:28.219846 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 00:12:28.219927 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:12:28.220055 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 00:12:28.222723 30991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: Bootstrap starting.
I20260812 00:12:28.223598 30991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:28.224201 30991 log.cc:824] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:28.225262 30991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: No bootstrap required, opened a new log
I20260812 00:12:28.228252 30991 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.228418 30991 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:28.228458 30991 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bfb53b6e5a4c698a7625da547f5c1f, State: Initialized, Role: FOLLOWER
I20260812 00:12:28.228585 30991 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.228961 30991 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:28.228930 31197 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.229136 31197 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:12:28.231251 31210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:28.231458 31210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:28.233825 30919 master_runner.cc:263] No leader master found from master a2bfb53b6e5a4c698a7625da547f5c1f
I20260812 00:12:28.233889 30919 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:12:28.235749 31167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:28.235868 31167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.236128 31167 leader_election.cc:288] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2bfb53b6e5a4c698a7625da547f5c1f (127.16.99.254:46183), 23c0ed05a4f4451198cdce1a2e9b0f31 (127.16.99.253:34975)
I20260812 00:12:28.236487 30964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71c27c3822c94478bd360eed55944f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" is_pre_election: true
I20260812 00:12:28.236671 30964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c27c3822c94478bd360eed55944f01 in term 0.
I20260812 00:12:28.237087 31099 leader_election.cc:302] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c27c3822c94478bd360eed55944f01, a2bfb53b6e5a4c698a7625da547f5c1f; no voters:
I20260812 00:12:28.237509 31046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71c27c3822c94478bd360eed55944f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" is_pre_election: true
I20260812 00:12:28.237672 31046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71c27c3822c94478bd360eed55944f01 in term 0.
I20260812 00:12:28.237999 31167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:28.238070 31167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:28.238096 31167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:28.239286 31167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.239430 31167 leader_election.cc:288] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [CANDIDATE]: Term 1 election: Requested vote from peers a2bfb53b6e5a4c698a7625da547f5c1f (127.16.99.254:46183), 23c0ed05a4f4451198cdce1a2e9b0f31 (127.16.99.253:34975)
I20260812 00:12:28.239889 30964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71c27c3822c94478bd360eed55944f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f"
I20260812 00:12:28.239980 30964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:28.241458 30964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c27c3822c94478bd360eed55944f01 in term 1.
I20260812 00:12:28.242574 31099 leader_election.cc:302] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71c27c3822c94478bd360eed55944f01, a2bfb53b6e5a4c698a7625da547f5c1f; no voters:
I20260812 00:12:28.243039 31046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71c27c3822c94478bd360eed55944f01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31"
I20260812 00:12:28.243132 31046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:28.244120 31046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71c27c3822c94478bd360eed55944f01 in term 1.
I20260812 00:12:28.244415 31167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:28.244601 31167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [term 1 LEADER]: Becoming Leader. State: Replica: 71c27c3822c94478bd360eed55944f01, State: Running, Role: LEADER
I20260812 00:12:28.244716 31167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } }
I20260812 00:12:28.245148 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c27c3822c94478bd360eed55944f01. Latest consensus state: current_term: 1 leader_uuid: "71c27c3822c94478bd360eed55944f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.245222 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:28.245594 31214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:28.246001 31214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:28.248948 31046 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [term 1 FOLLOWER]: Refusing update from remote peer 71c27c3822c94478bd360eed55944f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:28.249267 30964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [term 1 FOLLOWER]: Refusing update from remote peer 71c27c3822c94478bd360eed55944f01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:28.249794 31167 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:28.250332 31167 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:28.251937 31197 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c27c3822c94478bd360eed55944f01. Latest consensus state: current_term: 1 leader_uuid: "71c27c3822c94478bd360eed55944f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.252053 31197 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:28.253798 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c27c3822c94478bd360eed55944f01. Latest consensus state: current_term: 1 leader_uuid: "71c27c3822c94478bd360eed55944f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.253918 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:28.257797 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71c27c3822c94478bd360eed55944f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.257917 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: This master's current role is: LEADER
W20260812 00:12:28.258183 30942 tablet.cc:2369] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:28.258561 31181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71c27c3822c94478bd360eed55944f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.258663 31181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:28.259014 31215 mvcc.cc:205] Tried to move back new op lower bound from 7317477573623853056 to 7317477573611102208. Current Snapshot: MvccSnapshot[applied={T|T < 7317477573623853056}]
I20260812 00:12:28.259260 31218 mvcc.cc:205] Tried to move back new op lower bound from 7317477573623853056 to 7317477573611102208. Current Snapshot: MvccSnapshot[applied={T|T < 7317477573623853056}]
I20260812 00:12:28.260195 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71c27c3822c94478bd360eed55944f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.260298 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:28.261006 31197 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71c27c3822c94478bd360eed55944f01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfb53b6e5a4c698a7625da547f5c1f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 46183 } } peers { permanent_uuid: "23c0ed05a4f4451198cdce1a2e9b0f31" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 34975 } } peers { permanent_uuid: "71c27c3822c94478bd360eed55944f01" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 34679 } } }
I20260812 00:12:28.261106 31197 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:12:28.261509 31216 mvcc.cc:205] Tried to move back new op lower bound from 7317477573623853056 to 7317477573611102208. Current Snapshot: MvccSnapshot[applied={T|T < 7317477573623853056}]
I20260812 00:12:28.263417 31214 catalog_manager.cc:1347] Generated new cluster ID: 77e21e12ffb847219ab498bec6971025
I20260812 00:12:28.263471 31214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:28.291723 31214 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:28.292681 31214 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:28.305363 31214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c27c3822c94478bd360eed55944f01: Generated new TSK 0
I20260812 00:12:28.305785 31214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:29.201121 31193 catalog_manager.cc:1259] Loaded cluster ID: 77e21e12ffb847219ab498bec6971025
I20260812 00:12:29.201212 31193 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: loading cluster ID for follower catalog manager: success
I20260812 00:12:29.202270 31193 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: acquiring CA information for follower catalog manager: success
I20260812 00:12:29.202700 31193 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 23c0ed05a4f4451198cdce1a2e9b0f31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:12:29.232057 31210 catalog_manager.cc:1259] Loaded cluster ID: 77e21e12ffb847219ab498bec6971025
I20260812 00:12:29.232249 31210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: loading cluster ID for follower catalog manager: success
I20260812 00:12:29.233285 31210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: acquiring CA information for follower catalog manager: success
I20260812 00:12:29.233652 31210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2bfb53b6e5a4c698a7625da547f5c1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:12:29.380587 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30919
I20260812 00:12:29.388859 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 30998
I20260812 00:12:29.395761 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 31079
2026-08-12T00:12:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3052 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:29 dist-test-slave-zpfg krb5kdc[31232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:29 dist-test-slave-zpfg krb5kdc[31232](info): set up 2 sockets
Aug 12 00:12:29 dist-test-slave-zpfg krb5kdc[31232](info): commencing operation
krb5kdc: starting...
W20260812 00:12:30.515071 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:12:30 dist-test-slave-zpfg krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:12:30.653311 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:36071
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:34647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:36071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:30.740773 31247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:30.741043 31247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:30.741070 31247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:30.742599 31247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:30.742656 31247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:30.742672 31247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:30.742686 31247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:30.742699 31247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:30.744571 31247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:36071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:36071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:30.744915 31247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:30.745271 31247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:30.751075 31253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:30.751015 31255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:30.751204 31247 server_base.cc:1034] running on GCE node
W20260812 00:12:30.751016 31252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:30.751626 31247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:30.752008 31247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:30.753170 31247 hybrid_clock.cc:648] HybridClock initialized: now 1786493550753108 us; error 83 us; skew 500 ppm
Aug 12 00:12:30 dist-test-slave-zpfg krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493550, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:30.755894 31247 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:30.756523 31247 webserver.cc:460] Webserver started at http://127.16.99.254:38371/ using document root <none> and password file <none>
I20260812 00:12:30.756790 31247 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:30.756873 31247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:30.757020 31247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:30.758136 31247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "a64e4c3b9d514734a9a826a97ef11d26"
format_stamp: "Formatted at 2026-08-12 00:12:30 on dist-test-slave-zpfg"
server_key: "d10e32ab2876559db0fdf54152a36df7"
server_key_iv: "1ae19d086b985bd0d866101890c31702"
server_key_version: "encryptionkey@0"
I20260812 00:12:30.758623 31247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "a64e4c3b9d514734a9a826a97ef11d26"
format_stamp: "Formatted at 2026-08-12 00:12:30 on dist-test-slave-zpfg"
server_key: "d10e32ab2876559db0fdf54152a36df7"
server_key_iv: "1ae19d086b985bd0d866101890c31702"
server_key_version: "encryptionkey@0"
I20260812 00:12:30.760113 31247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:30.761021 31262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:30.761222 31247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:30.761380 31247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "a64e4c3b9d514734a9a826a97ef11d26"
format_stamp: "Formatted at 2026-08-12 00:12:30 on dist-test-slave-zpfg"
server_key: "d10e32ab2876559db0fdf54152a36df7"
server_key_iv: "1ae19d086b985bd0d866101890c31702"
server_key_version: "encryptionkey@0"
I20260812 00:12:30.761469 31247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:30.775656 31247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:30.779784 31247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:30.779970 31247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:30.786129 31247 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:36071
I20260812 00:12:30.786194 31317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:36071 every 8 connection(s)
I20260812 00:12:30.786755 31247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:30.788007 31318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:30.789592 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 31247
I20260812 00:12:30.789732 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:12:30.789963 16783 external_mini_cluster.cc:1428] Setting key fb241881025c7fb79ad7df6b788947dd
I20260812 00:12:30.790881 31318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26: Bootstrap starting.
I20260812 00:12:30.791646 31318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:30.791996 31318 log.cc:824] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:30.792888 31318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26: No bootstrap required, opened a new log
I20260812 00:12:30.795073 31318 raft_consensus.cc:348] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e4c3b9d514734a9a826a97ef11d26" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36071 } }
I20260812 00:12:30.795271 31318 raft_consensus.cc:374] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:30.795310 31318 raft_consensus.cc:729] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64e4c3b9d514734a9a826a97ef11d26, State: Initialized, Role: FOLLOWER
I20260812 00:12:30.795471 31318 consensus_queue.cc:260] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e4c3b9d514734a9a826a97ef11d26" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36071 } }
I20260812 00:12:30.795555 31318 raft_consensus.cc:388] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:30.795608 31318 raft_consensus.cc:482] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:30.795650 31318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:30.796463 31318 raft_consensus.cc:504] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e4c3b9d514734a9a826a97ef11d26" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36071 } }
I20260812 00:12:30.796598 31318 leader_election.cc:302] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a64e4c3b9d514734a9a826a97ef11d26; no voters:
I20260812 00:12:30.796819 31318 leader_election.cc:288] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:30.796851 31324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:30.796972 31324 raft_consensus.cc:686] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [term 1 LEADER]: Becoming Leader. State: Replica: a64e4c3b9d514734a9a826a97ef11d26, State: Running, Role: LEADER
I20260812 00:12:30.797060 31324 consensus_queue.cc:237] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e4c3b9d514734a9a826a97ef11d26" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36071 } }
I20260812 00:12:30.797103 31318 sys_catalog.cc:564] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:30.797382 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64e4c3b9d514734a9a826a97ef11d26. Latest consensus state: current_term: 1 leader_uuid: "a64e4c3b9d514734a9a826a97ef11d26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e4c3b9d514734a9a826a97ef11d26" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36071 } } }
I20260812 00:12:30.797559 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:30.797399 31326 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a64e4c3b9d514734a9a826a97ef11d26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64e4c3b9d514734a9a826a97ef11d26" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 36071 } } }
I20260812 00:12:30.797886 31326 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:30.798079 31331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 00:12:30 dist-test-slave-zpfg krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:30.798678 31331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:30.800627 31331 catalog_manager.cc:1347] Generated new cluster ID: c6edb2a1ce384aaa80c8d512103d270c
I20260812 00:12:30.800688 31331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:30.804001 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:30.790676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.16.99.254:36071)
0812 00:12:30.790903 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:30.790914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:30.790952 (+ 38us) server_negotiation.cc:407] Connection header received
0812 00:12:30.791579 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:30.791590 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:30.791661 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:30.791851 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:30.792785 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.794014 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.795207 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:30.795850 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:30.795943 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:30.799371 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:30.799378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:30.799378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:30.799393 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:30.802151 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:30.802667 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:30.802671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:30.802671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:30.802777 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:30.803019 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:30.803020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:30.803020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:30.803317 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:30.803466 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:30.803597 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 00:12:30.803735 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":106,"threads_started":1}
I20260812 00:12:30.804764 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:12:30.810509 31331 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:30.811148 31331 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:30.816803 31331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a64e4c3b9d514734a9a826a97ef11d26: Generated new TSK 0
I20260812 00:12:30.817044 31331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:12:30.859576 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 31247
2026-08-12T00:12:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1462 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:30 dist-test-slave-zpfg krb5kdc[31347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:30 dist-test-slave-zpfg krb5kdc[31347](info): set up 2 sockets
Aug 12 00:12:30 dist-test-slave-zpfg krb5kdc[31347](info): commencing operation
krb5kdc: starting...
W20260812 00:12:31.983559 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:12:32.135198 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42197
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:32.230973 31361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:32.231285 31361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:32.231333 31361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:32.232673 31361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:32.232771 31361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:32.232796 31361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:32.232818 31361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:32.232846 31361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:32.234587 31361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:42197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:32.234944 31361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:32.235234 31361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:32.240886 31369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:32.240891 31367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:32.241284 31361 server_base.cc:1034] running on GCE node
W20260812 00:12:32.240888 31366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:32.241585 31361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:32.241948 31361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:32.243161 31361 hybrid_clock.cc:648] HybridClock initialized: now 1786493552243136 us; error 44 us; skew 500 ppm
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:32.246810 31361 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:32.247390 31361 webserver.cc:460] Webserver started at http://127.16.99.254:32927/ using document root <none> and password file <none>
I20260812 00:12:32.247641 31361 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:32.247720 31361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:32.247879 31361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:32.249091 31361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "664dae6d3d0943e19634717ec4e72d2e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "6c2701a08dc1724eb9a2e8fc1ee97229"
server_key_iv: "ec909bef0be98c2dfe7668c445b09b58"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.249513 31361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "664dae6d3d0943e19634717ec4e72d2e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "6c2701a08dc1724eb9a2e8fc1ee97229"
server_key_iv: "ec909bef0be98c2dfe7668c445b09b58"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.251119 31361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:32.252261 31377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.252529 31361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:32.252655 31361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "664dae6d3d0943e19634717ec4e72d2e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "6c2701a08dc1724eb9a2e8fc1ee97229"
server_key_iv: "ec909bef0be98c2dfe7668c445b09b58"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.252745 31361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:32.267417 31361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:32.270102 31361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:32.270444 31361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:32.275040 31361 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:42197
I20260812 00:12:32.275092 31432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:42197 every 8 connection(s)
I20260812 00:12:32.275530 31361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:32.276710 31433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:32.279810 31433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Bootstrap starting.
I20260812 00:12:32.280611 31433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:32.281030 31433 log.cc:824] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:32.282023 31433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: No bootstrap required, opened a new log
I20260812 00:12:32.282660 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 31361
I20260812 00:12:32.282794 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:12:32.283013 16783 external_mini_cluster.cc:1428] Setting key 460d2b8aa7eb58649388c2d634c35803
I20260812 00:12:32.284446 31433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:32.284638 31433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:32.284722 31433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664dae6d3d0943e19634717ec4e72d2e, State: Initialized, Role: FOLLOWER
I20260812 00:12:32.284989 31433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:32.285118 31433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:32.285167 31433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:32.285213 31433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:32.286404 31433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:32.286532 31433 leader_election.cc:302] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 664dae6d3d0943e19634717ec4e72d2e; no voters:
I20260812 00:12:32.286798 31433 leader_election.cc:288] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:32.286839 31438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:32.287045 31438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 LEADER]: Becoming Leader. State: Replica: 664dae6d3d0943e19634717ec4e72d2e, State: Running, Role: LEADER
I20260812 00:12:32.287108 31433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:32.287189 31438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:32.287629 31438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 664dae6d3d0943e19634717ec4e72d2e. Latest consensus state: current_term: 1 leader_uuid: "664dae6d3d0943e19634717ec4e72d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } } }
I20260812 00:12:32.287643 31439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "664dae6d3d0943e19634717ec4e72d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } } }
I20260812 00:12:32.287786 31439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:32.287992 31438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: This master's current role is: LEADER
W20260812 00:12:32.289366 31454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:12:32.289465 31454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:12:32.289572 31445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:32.289775 31445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:32.291730 31445 catalog_manager.cc:1347] Generated new cluster ID: a7639f61e0fb44e49e12b9ea008ab71d
I20260812 00:12:32.291779 31445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:32.296703 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.283666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.16.99.254:42197)
0812 00:12:32.283876 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.283885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.283921 (+ 36us) server_negotiation.cc:407] Connection header received
0812 00:12:32.284443 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.284453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.284511 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.284641 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.285707 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.286876 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.288264 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.288850 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.288965 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.292177 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:32.292181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:32.292182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:32.292196 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:32.294641 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.295098 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.295103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.295103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.295270 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.295502 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.295503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.295503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.295858 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:32.296048 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.296221 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.296430 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":115,"threads_started":1}
I20260812 00:12:32.312654 31445 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:32.313462 31445 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:32.322615 31445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Generated new TSK 0
I20260812 00:12:32.322961 31445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:12:32.361783 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42197
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:32.460222 31460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:32.460482 31460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:32.460530 31460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:32.461894 31460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:32.461972 31460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:32.462023 31460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:12:32.463816 31460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:32.464143 31460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:32.464425 31460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:32.470213 31468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:32.470379 31465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:32.470396 31466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:32.470664 31460 server_base.cc:1034] running on GCE node
I20260812 00:12:32.470868 31460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:32.471164 31460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:32.472343 31460 hybrid_clock.cc:648] HybridClock initialized: now 1786493552472329 us; error 33 us; skew 500 ppm
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:32.475914 31460 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:12:32.476600 31460 webserver.cc:460] Webserver started at http://127.16.99.193:32789/ using document root <none> and password file <none>
I20260812 00:12:32.476897 31460 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:32.476987 31460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:32.477159 31460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:32.478463 31460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "1b796088d1774259abebdbb7ce7bbab5"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "7584c7a8e4bebef678f31b9fc6f021d6"
server_key_iv: "32317a90de55c9391ba33cb9023f09f9"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.478892 31460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "1b796088d1774259abebdbb7ce7bbab5"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "7584c7a8e4bebef678f31b9fc6f021d6"
server_key_iv: "32317a90de55c9391ba33cb9023f09f9"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.480427 31460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:12:32.481519 31475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.481812 31460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:32.481949 31460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "1b796088d1774259abebdbb7ce7bbab5"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "7584c7a8e4bebef678f31b9fc6f021d6"
server_key_iv: "32317a90de55c9391ba33cb9023f09f9"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.482015 31460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:32.492694 31460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:32.495245 31460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:32.495369 31460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:32.495585 31460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:32.495935 31460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:32.495999 31460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.496037 31460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:32.496079 31460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.503024 31460 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:45483
I20260812 00:12:32.503075 31595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:45483 every 8 connection(s)
I20260812 00:12:32.503433 31460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:32.509027 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 31460
I20260812 00:12:32.509155 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:32.509441 16783 external_mini_cluster.cc:1428] Setting key 5faeed82ce9494dc52d931b5ecda0bfc
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:32.513917 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.504227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:52087 (local address 127.16.99.254:42197)
0812 00:12:32.504371 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.504374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.504471 (+ 97us) server_negotiation.cc:407] Connection header received
0812 00:12:32.504958 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.504959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.504974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.505030 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.506031 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.506646 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.507713 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.507952 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.508012 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.510828 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:32.510829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:32.510829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:32.510850 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:32.512397 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.512876 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.512877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.512878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.512941 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.513231 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.513232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.513233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.513357 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:32.513434 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.513722 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.513820 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 00:12:32.514135 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.504163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:52087)
0812 00:12:32.504341 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.504381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.504870 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.505086 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.505089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.505715 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:32.505926 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.505932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.506762 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.506763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.507613 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.507616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.508040 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.508041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.508169 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.508825 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:32.508834 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:32.510716 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:32.512512 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:32.512522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:32.512524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:32.512792 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:32.513026 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:32.513027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:32.513027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:32.513155 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:32.513424 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:32.513432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:32.513587 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 00:12:32.513702 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 00:12:32.513876 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260812 00:12:32.514698 31596 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42197
I20260812 00:12:32.514798 31596 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:32.515024 31596 heartbeater.cc:511] Master 127.16.99.254:42197 requested a full tablet report, sending...
I20260812 00:12:32.515614 31397 ts_manager.cc:194] Registered new tserver with Master: 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193:45483)
I20260812 00:12:32.516260 31397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:52087
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:12:32.542253 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42197
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:32.636734 31602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:32.636982 31602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:32.637012 31602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:32.638504 31602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:32.638561 31602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:32.638593 31602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:12:32.640326 31602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:32.640621 31602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:32.640911 31602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:32.646644 31610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:32.646715 31608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:32.646661 31607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:32.646883 31602 server_base.cc:1034] running on GCE node
I20260812 00:12:32.647123 31602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:32.647358 31602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:32.648507 31602 hybrid_clock.cc:648] HybridClock initialized: now 1786493552648484 us; error 39 us; skew 500 ppm
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:32.651600 31602 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:12:32.652190 31602 webserver.cc:460] Webserver started at http://127.16.99.194:46777/ using document root <none> and password file <none>
I20260812 00:12:32.652490 31602 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:32.652571 31602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:32.652779 31602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:32.654089 31602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "288059d2de25484ead025e1137e916fd"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "ea701dc3ea580ecf5b6455757e969dd1"
server_key_iv: "222a898547bfdbbc9299552cbd2268bd"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.654553 31602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "288059d2de25484ead025e1137e916fd"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "ea701dc3ea580ecf5b6455757e969dd1"
server_key_iv: "222a898547bfdbbc9299552cbd2268bd"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.656355 31602 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260812 00:12:32.657348 31617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.657558 31602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:32.657707 31602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "288059d2de25484ead025e1137e916fd"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "ea701dc3ea580ecf5b6455757e969dd1"
server_key_iv: "222a898547bfdbbc9299552cbd2268bd"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.657797 31602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:32.672856 31602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:32.675531 31602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:32.675715 31602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:32.675979 31602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:32.676333 31602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:32.676394 31602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.676438 31602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:32.676463 31602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.683012 31602 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:37301
I20260812 00:12:32.683059 31736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:37301 every 8 connection(s)
I20260812 00:12:32.683409 31602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:32.689244 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 31602
I20260812 00:12:32.689396 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:32.689594 16783 external_mini_cluster.cc:1428] Setting key c05a37e9c07224e5714e7f5f54bcb7fb
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:32.694756 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.684220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:32895 (local address 127.16.99.254:42197)
0812 00:12:32.684295 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.684298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.684556 (+ 258us) server_negotiation.cc:407] Connection header received
0812 00:12:32.685084 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.685084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.685095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.685295 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.686353 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.687051 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.688313 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.688578 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.688639 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.691274 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:32.691276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:32.691276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:32.691286 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:32.693130 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.693650 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.693651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.693651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.693709 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.694035 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.694036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.694036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.694157 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:32.694214 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.694609 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.694687 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:12:32.695248 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.684198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:32895)
0812 00:12:32.684416 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.684456 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.684989 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.685243 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.685247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.685966 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:32.686180 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.686186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.687157 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.687158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.688204 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.688207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.688660 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.688661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.688768 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.689564 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:32.689572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:32.691167 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:32.693287 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:32.693297 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:32.693299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:32.693556 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:32.693788 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:32.693789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:32.693790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:32.693954 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:32.694230 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:32.694237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:32.694455 (+ 218us) client_negotiation.cc:769] Sending connection context
0812 00:12:32.694583 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 00:12:32.694807 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":123,"threads_started":1}
I20260812 00:12:32.695977 31737 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42197
I20260812 00:12:32.696123 31737 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:32.696389 31737 heartbeater.cc:511] Master 127.16.99.254:42197 requested a full tablet report, sending...
I20260812 00:12:32.696919 31397 ts_manager.cc:194] Registered new tserver with Master: 288059d2de25484ead025e1137e916fd (127.16.99.194:37301)
I20260812 00:12:32.697319 31397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:32895
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:12:32.722853 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42197
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:32.820672 31742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:32.821003 31742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:32.821071 31742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:32.822811 31742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:32.822898 31742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:32.822954 31742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:12:32.824931 31742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:32.825299 31742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:32.825591 31742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:32.831678 31747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:32.831677 31750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:32.831801 31748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:32.832082 31742 server_base.cc:1034] running on GCE node
I20260812 00:12:32.832515 31742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:32.832839 31742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:32.834210 31742 hybrid_clock.cc:648] HybridClock initialized: now 1786493552834087 us; error 55 us; skew 500 ppm
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:32.837719 31742 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:12:32.838579 31742 webserver.cc:460] Webserver started at http://127.16.99.195:44443/ using document root <none> and password file <none>
I20260812 00:12:32.838879 31742 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:32.838965 31742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:32.839154 31742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:32.840448 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "1f1fec17f2e446c98f6a39bc40437a8e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "5678630ae2e907205a244a5d3fdabef8"
server_key_iv: "beff7880171ce41c21e85767339ada17"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.840876 31742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "1f1fec17f2e446c98f6a39bc40437a8e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "5678630ae2e907205a244a5d3fdabef8"
server_key_iv: "beff7880171ce41c21e85767339ada17"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.842700 31742 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260812 00:12:32.843739 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.843999 31742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:32.844249 31742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "1f1fec17f2e446c98f6a39bc40437a8e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "5678630ae2e907205a244a5d3fdabef8"
server_key_iv: "beff7880171ce41c21e85767339ada17"
server_key_version: "encryptionkey@0"
I20260812 00:12:32.844375 31742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:32.862083 31742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:32.864642 31742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:32.864848 31742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:32.865128 31742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:32.865521 31742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:32.865559 31742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.865597 31742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:32.865653 31742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:32.872555 31742 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:34447
I20260812 00:12:32.872648 31883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:34447 every 8 connection(s)
I20260812 00:12:32.873059 31742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:32.873945 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 31742
I20260812 00:12:32.874056 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:32.874264 16783 external_mini_cluster.cc:1428] Setting key 7c524920c8c32d0a700e607715f094d2
Aug 12 00:12:32 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493552, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:32.884126 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.874060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:48355 (local address 127.16.99.254:42197)
0812 00:12:32.874119 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.874122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.874472 (+ 350us) server_negotiation.cc:407] Connection header received
0812 00:12:32.875062 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.875064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.875077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.875136 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.876229 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.876831 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.877878 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.878085 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.878166 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.880879 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:32.880881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:32.880881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:32.880890 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:32.882486 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.883030 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.883032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.883032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.883097 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.883403 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.883405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.883405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.883538 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:32.883615 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.883934 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.884029 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:12:32.884353 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.874064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:48355)
0812 00:12:32.874289 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.874368 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.874953 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.875215 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.875219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.875888 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:32.876119 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.876125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.876946 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.876947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.877776 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.877778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.878190 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.878191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.878331 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.878991 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:32.878999 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:32.880611 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:32.882599 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:32.882611 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:32.882614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:32.882926 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:32.883190 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:32.883191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:32.883191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:32.883332 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:32.883616 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:32.883624 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:32.883787 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 00:12:32.883904 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:12:32.884089 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260812 00:12:32.884919 31884 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42197
I20260812 00:12:32.885030 31884 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:32.885250 31884 heartbeater.cc:511] Master 127.16.99.254:42197 requested a full tablet report, sending...
I20260812 00:12:32.886240 31396 ts_manager.cc:194] Registered new tserver with Master: 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447)
I20260812 00:12:32.886897 31396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:48355
I20260812 00:12:32.887104 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:32.896627 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.889043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.16.99.254:42197)
0812 00:12:32.889127 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.889129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.889265 (+ 136us) server_negotiation.cc:407] Connection header received
0812 00:12:32.889360 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.889361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.889373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.889470 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:32.890188 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.890855 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.891757 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.891994 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.892096 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.893607 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:32.893609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:32.893609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:32.893618 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:32.895224 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.895628 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.895630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.895630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.895699 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:32.895924 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:32.895925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:32.895925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:32.896134 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:32.896261 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.896402 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.896512 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 00:12:32.899462 31396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:32.901261 31396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:32.910166 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.905786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.16.99.194:37301)
0812 00:12:32.906067 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.906073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.906082 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:12:32.906120 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.906122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.906137 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.906516 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:32.906598 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.907307 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.908968 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.909662 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.909795 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.909871 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.910006 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.910066 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":127,"threads_started":1}
I20260812 00:12:32.910151 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.904549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37301 (local address 127.0.0.1:34218)
0812 00:12:32.904777 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.904826 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.904908 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.906235 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.906236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.906238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:32.906429 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.906432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.907432 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.907433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.908849 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.908853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.909791 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.909792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.909887 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.909903 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 00:12:32.909974 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 00:12:32.910048 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":423,"threads_started":1}
I20260812 00:12:32.910539 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.904563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45483 (local address 127.0.0.1:58698)
0812 00:12:32.905825 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.905837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.905917 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.906176 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.906181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.906183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:32.906457 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.906460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.907949 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.907950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.908947 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.908950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.910165 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.910167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.910280 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.910287 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 00:12:32.910332 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 00:12:32.910438 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"mutex_wait_us":72,"thread_start_us":84,"threads_started":1}
I20260812 00:12:32.911079 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.904792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.16.99.193:45483)
0812 00:12:32.905050 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.905056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.905940 (+ 884us) server_negotiation.cc:407] Connection header received
0812 00:12:32.906002 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.906005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.906025 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.906141 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:32.906572 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.907811 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.909119 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.910061 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.910807 (+ 746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.910893 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.910928 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.910983 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260812 00:12:32.911266 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.905276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44846 (local address 127.16.99.195:34447)
0812 00:12:32.905566 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:32.905571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:32.905926 (+ 355us) server_negotiation.cc:407] Connection header received
0812 00:12:32.906058 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:32.906061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:32.906080 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:32.906140 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:32.906502 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.907956 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.909296 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.910862 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.910940 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.911000 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:32.911137 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 00:12:32.911184 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":203,"threads_started":1}
I20260812 00:12:32.911429 31667 tablet_service.cc:1467] Processing CreateTablet for tablet f01145e1c2c34b9cb73126e330d94c72 (DEFAULT_TABLE table=test-table [id=62432c1fa5c644c9868325f9c14007c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:32.911566 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:32.904549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:34447 (local address 127.0.0.1:44846)
0812 00:12:32.905855 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:32.905865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:32.905955 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:32.906262 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:32.906262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:32.906264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:32.906428 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.906432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.908083 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.908083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.909199 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:32.909202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:32.910964 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:32.910965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:32.911033 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:32.911038 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:32.911109 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 00:12:32.911478 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"mutex_wait_us":1115,"thread_start_us":94,"threads_started":1}
I20260812 00:12:32.911862 31667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01145e1c2c34b9cb73126e330d94c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:32.911891 31527 tablet_service.cc:1467] Processing CreateTablet for tablet f01145e1c2c34b9cb73126e330d94c72 (DEFAULT_TABLE table=test-table [id=62432c1fa5c644c9868325f9c14007c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:32.912298 31527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01145e1c2c34b9cb73126e330d94c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:32.912333 31811 tablet_service.cc:1467] Processing CreateTablet for tablet f01145e1c2c34b9cb73126e330d94c72 (DEFAULT_TABLE table=test-table [id=62432c1fa5c644c9868325f9c14007c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:32.912730 31811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01145e1c2c34b9cb73126e330d94c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:32.915527 31903 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Bootstrap starting.
I20260812 00:12:32.916304 31904 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap starting.
I20260812 00:12:32.916368 31903 tablet_bootstrap.cc:654] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:32.916638 31905 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Bootstrap starting.
I20260812 00:12:32.917130 31904 tablet_bootstrap.cc:654] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:32.917279 31903 log.cc:824] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:32.917438 31905 tablet_bootstrap.cc:654] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:32.917744 31904 log.cc:824] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:32.918164 31905 log.cc:824] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:32.918381 31903 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: No bootstrap required, opened a new log
I20260812 00:12:32.918473 31903 ts_tablet_manager.cc:1397] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:12:32.918730 31904 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: No bootstrap required, opened a new log
I20260812 00:12:32.918797 31904 ts_tablet_manager.cc:1397] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:12:32.919138 31905 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: No bootstrap required, opened a new log
I20260812 00:12:32.919248 31905 ts_tablet_manager.cc:1397] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:12:32.920703 31903 raft_consensus.cc:348] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:32.920920 31903 raft_consensus.cc:374] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:32.920964 31903 raft_consensus.cc:729] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288059d2de25484ead025e1137e916fd, State: Initialized, Role: FOLLOWER
I20260812 00:12:32.920985 31904 raft_consensus.cc:348] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:32.921123 31903 consensus_queue.cc:260] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:32.921200 31904 raft_consensus.cc:374] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:32.921240 31904 raft_consensus.cc:729] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b796088d1774259abebdbb7ce7bbab5, State: Initialized, Role: FOLLOWER
I20260812 00:12:32.921268 31905 raft_consensus.cc:348] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:32.921391 31904 consensus_queue.cc:260] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:32.921469 31903 ts_tablet_manager.cc:1428] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:32.921492 31905 raft_consensus.cc:374] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:32.921550 31905 raft_consensus.cc:729] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1fec17f2e446c98f6a39bc40437a8e, State: Initialized, Role: FOLLOWER
I20260812 00:12:32.921598 31737 heartbeater.cc:503] Master 127.16.99.254:42197 was elected leader, sending a full tablet report...
I20260812 00:12:32.921693 31905 consensus_queue.cc:260] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:32.921762 31904 ts_tablet_manager.cc:1428] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:12:32.921891 31596 heartbeater.cc:503] Master 127.16.99.254:42197 was elected leader, sending a full tablet report...
I20260812 00:12:32.922056 31905 ts_tablet_manager.cc:1428] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:32.922111 31884 heartbeater.cc:503] Master 127.16.99.254:42197 was elected leader, sending a full tablet report...
W20260812 00:12:32.934134 31738 tablet.cc:2369] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:12:33.004499 31597 tablet.cc:2369] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:33.022142 31910 raft_consensus.cc:482] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:33.022287 31910 raft_consensus.cc:504] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.022790 31910 leader_election.cc:288] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447), 288059d2de25484ead025e1137e916fd (127.16.99.194:37301)
I20260812 00:12:33.027271 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.023213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:55857 (local address 127.16.99.195:34447)
0812 00:12:33.023303 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.023306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.023495 (+ 189us) server_negotiation.cc:407] Connection header received
0812 00:12:33.023550 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.023551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.023562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.023630 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:33.023958 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.024689 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.026208 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.026856 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.026927 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.026935 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.027143 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.027187 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 00:12:33.027271 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.023239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:54255 (local address 127.16.99.194:37301)
0812 00:12:33.023311 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.023314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.023325 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:33.023450 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.023452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.023464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.023531 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:33.023800 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.024770 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.026154 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.026805 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.026895 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.026907 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.027140 (+ 233us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.027187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:12:33.027472 31912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.023060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37301 (local address 127.16.99.193:54255)
0812 00:12:33.023215 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.023229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.023357 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.023548 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.023549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.023552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:33.023643 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.023646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.024849 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.024851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.026041 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.026049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.026944 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.026944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.027037 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.027046 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.027119 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.027320 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":40,"thread_start_us":70,"threads_started":1}
I20260812 00:12:33.027498 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.023060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:34447 (local address 127.16.99.193:55857)
0812 00:12:33.023365 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.023376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.023480 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.023760 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.023762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.023767 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:33.023859 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.023860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.024825 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.024826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.026065 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.026069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.026940 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.026942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.027037 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.027048 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.027119 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.027320 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":86,"thread_start_us":99,"threads_started":1}
I20260812 00:12:33.028090 31689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01145e1c2c34b9cb73126e330d94c72" candidate_uuid: "1b796088d1774259abebdbb7ce7bbab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288059d2de25484ead025e1137e916fd" is_pre_election: true
I20260812 00:12:33.028090 31832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01145e1c2c34b9cb73126e330d94c72" candidate_uuid: "1b796088d1774259abebdbb7ce7bbab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" is_pre_election: true
I20260812 00:12:33.028308 31689 raft_consensus.cc:2454] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b796088d1774259abebdbb7ce7bbab5 in term 0.
I20260812 00:12:33.028308 31832 raft_consensus.cc:2454] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b796088d1774259abebdbb7ce7bbab5 in term 0.
I20260812 00:12:33.028590 31477 leader_election.cc:302] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5, 1f1fec17f2e446c98f6a39bc40437a8e; no voters:
I20260812 00:12:33.028800 31910 raft_consensus.cc:2781] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:33.028900 31910 raft_consensus.cc:482] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:33.028944 31910 raft_consensus.cc:3037] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:33.029839 31910 raft_consensus.cc:504] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.030069 31910 leader_election.cc:288] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 1 election: Requested vote from peers 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447), 288059d2de25484ead025e1137e916fd (127.16.99.194:37301)
I20260812 00:12:33.030333 31689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01145e1c2c34b9cb73126e330d94c72" candidate_uuid: "1b796088d1774259abebdbb7ce7bbab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288059d2de25484ead025e1137e916fd"
I20260812 00:12:33.030488 31689 raft_consensus.cc:3037] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:33.031338 31689 raft_consensus.cc:2454] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b796088d1774259abebdbb7ce7bbab5 in term 1.
I20260812 00:12:33.031517 31832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01145e1c2c34b9cb73126e330d94c72" candidate_uuid: "1b796088d1774259abebdbb7ce7bbab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1fec17f2e446c98f6a39bc40437a8e"
I20260812 00:12:33.031662 31479 leader_election.cc:302] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5, 288059d2de25484ead025e1137e916fd; no voters:
I20260812 00:12:33.031656 31832 raft_consensus.cc:3037] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:33.031770 31910 raft_consensus.cc:2781] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:33.031985 31910 raft_consensus.cc:686] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 1 LEADER]: Becoming Leader. State: Replica: 1b796088d1774259abebdbb7ce7bbab5, State: Running, Role: LEADER
I20260812 00:12:33.032233 31910 consensus_queue.cc:237] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.032764 31832 raft_consensus.cc:2454] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b796088d1774259abebdbb7ce7bbab5 in term 1.
I20260812 00:12:33.033381 31397 catalog_manager.cc:5495] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193). New cstate: current_term: 1 leader_uuid: "1b796088d1774259abebdbb7ce7bbab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } health_report { overall_health: HEALTHY } } }
I20260812 00:12:33.061718 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.058190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.16.99.193:45483)
0812 00:12:33.058269 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.058272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.058283 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:33.058390 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.058392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.058418 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.058500 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:33.058818 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.059375 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.060386 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.060600 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.060679 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.061052 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:33.061208 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:33.061315 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.061544 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.061576 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:12:33.064630 31689 raft_consensus.cc:1261] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Refusing update from remote peer 1b796088d1774259abebdbb7ce7bbab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:33.064630 31832 raft_consensus.cc:1261] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 FOLLOWER]: Refusing update from remote peer 1b796088d1774259abebdbb7ce7bbab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:33.065155 31910 consensus_queue.cc:1035] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:33.065382 31914 consensus_queue.cc:1035] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:33.068431 31917 mvcc.cc:205] Tried to move back new op lower bound from 7317477593349505024 to 7317477593220931584. Current Snapshot: MvccSnapshot[applied={T|T < 7317477593349505024}]
I20260812 00:12:33.068951 31919 mvcc.cc:205] Tried to move back new op lower bound from 7317477593349505024 to 7317477593220931584. Current Snapshot: MvccSnapshot[applied={T|T < 7317477593349505024}]
I20260812 00:12:33.069094 31918 mvcc.cc:205] Tried to move back new op lower bound from 7317477593349505024 to 7317477593220931584. Current Snapshot: MvccSnapshot[applied={T|T < 7317477593349505024}]
I20260812 00:12:33.071345 31397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:33.071457 31397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:33.073266 31397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Sending DeleteTablet for 3 replicas of tablet f01145e1c2c34b9cb73126e330d94c72
I20260812 00:12:33.073688 31667 tablet_service.cc:1514] Processing DeleteTablet for tablet f01145e1c2c34b9cb73126e330d94c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:33 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34218
I20260812 00:12:33.073750 31811 tablet_service.cc:1514] Processing DeleteTablet for tablet f01145e1c2c34b9cb73126e330d94c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:33 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:44846
I20260812 00:12:33.073990 31927 tablet_replica.cc:330] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: stopping tablet replica
I20260812 00:12:33.074146 31527 tablet_service.cc:1514] Processing DeleteTablet for tablet f01145e1c2c34b9cb73126e330d94c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:33 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:58698
I20260812 00:12:33.074177 31927 raft_consensus.cc:2229] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:33.074203 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 31460
I20260812 00:12:33.074319 31927 raft_consensus.cc:2258] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:33.074508 31926 tablet_replica.cc:330] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: stopping tablet replica
I20260812 00:12:33.074627 31926 raft_consensus.cc:2229] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:33.075191 31926 raft_consensus.cc:2258] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:33.076174 31927 ts_tablet_manager.cc:1905] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:33.077953 31927 ts_tablet_manager.cc:1918] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:33.078033 31927 log.cc:1196] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/f01145e1c2c34b9cb73126e330d94c72
I20260812 00:12:33.078501 31927 ts_tablet_manager.cc:1939] T f01145e1c2c34b9cb73126e330d94c72 P 288059d2de25484ead025e1137e916fd: Deleting consensus metadata
I20260812 00:12:33.079378 31381 catalog_manager.cc:4841] TS 288059d2de25484ead025e1137e916fd (127.16.99.194:37301): tablet f01145e1c2c34b9cb73126e330d94c72 (table test-table [id=62432c1fa5c644c9868325f9c14007c5]) successfully deleted
I20260812 00:12:33.079417 31926 ts_tablet_manager.cc:1905] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:33.081552 31926 ts_tablet_manager.cc:1918] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:33.081662 31926 log.cc:1196] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/f01145e1c2c34b9cb73126e330d94c72
I20260812 00:12:33.082006 31926 ts_tablet_manager.cc:1939] T f01145e1c2c34b9cb73126e330d94c72 P 1f1fec17f2e446c98f6a39bc40437a8e: Deleting consensus metadata
I20260812 00:12:33.082643 31379 catalog_manager.cc:4841] TS 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447): tablet f01145e1c2c34b9cb73126e330d94c72 (table test-table [id=62432c1fa5c644c9868325f9c14007c5]) successfully deleted
W20260812 00:12:33.084153 31378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:45483 (error 108)
I20260812 00:12:33.084501 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 31602
I20260812 00:12:33.085523 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.084651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45483 (local address 127.0.0.1:58714)
0812 00:12:33.085407 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.085473 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":729}
W20260812 00:12:33.086618 31378 catalog_manager.cc:4584] TS 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193:45483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f01145e1c2c34b9cb73126e330d94c72: Network error: Client connection negotiation failed: client connection to 127.16.99.193:45483: connect: Connection refused (error 111)
I20260812 00:12:33.096601 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 31742
I20260812 00:12:33.104638 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 31361
I20260812 00:12:33.113652 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42197
--webserver_interface=127.16.99.254
--webserver_port=32927
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:33.206151 31929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:33.206462 31929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:33.206491 31929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:33.208130 31929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:33.208199 31929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:33.208216 31929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:33.208231 31929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:33.208244 31929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:33.210198 31929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:42197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:33.210563 31929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:33.210883 31929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:33.224519 31938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:33.224573 31929 server_base.cc:1034] running on GCE node
W20260812 00:12:33.224519 31935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:33.224521 31936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:33.225302 31929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:33.225814 31929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:33.227144 31929 hybrid_clock.cc:648] HybridClock initialized: now 1786493553227073 us; error 43 us; skew 500 ppm
Aug 12 00:12:33 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:33.231050 31929 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:33.231691 31929 webserver.cc:460] Webserver started at http://127.16.99.254:32927/ using document root <none> and password file <none>
I20260812 00:12:33.232018 31929 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:33.232089 31929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:33.234246 31929 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:33.235378 31945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.235635 31929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:33.235737 31929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "664dae6d3d0943e19634717ec4e72d2e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "6c2701a08dc1724eb9a2e8fc1ee97229"
server_key_iv: "ec909bef0be98c2dfe7668c445b09b58"
server_key_version: "encryptionkey@0"
I20260812 00:12:33.236279 31929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:33.267550 31929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:33.270362 31929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:33.270557 31929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:33.275538 31929 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:42197
I20260812 00:12:33.275578 32002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:42197 every 8 connection(s)
I20260812 00:12:33.276036 31929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:33.278021 32003 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:12:33.279119 32003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Bootstrap starting.
I20260812 00:12:33.283685 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 31929
I20260812 00:12:33.284039 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:45483
--local_ip_for_outbound_sockets=127.16.99.193
--tserver_master_addrs=127.16.99.254:42197
--webserver_port=32789
--webserver_interface=127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:33.289558 32003 log.cc:824] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:33.292028 32003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:33.292436 32003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Bootstrap complete.
I20260812 00:12:33.295296 32003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:33.295612 32003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664dae6d3d0943e19634717ec4e72d2e, State: Initialized, Role: FOLLOWER
I20260812 00:12:33.295878 32003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:33.295996 32003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:33.296082 32003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:33.296144 32003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:12:33.297390 32003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:33.297544 32003 leader_election.cc:302] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 664dae6d3d0943e19634717ec4e72d2e; no voters:
I20260812 00:12:33.297807 32003 leader_election.cc:288] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:12:33.297853 32007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:12:33.297982 32007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 LEADER]: Becoming Leader. State: Replica: 664dae6d3d0943e19634717ec4e72d2e, State: Running, Role: LEADER
I20260812 00:12:33.298174 32003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:33.298174 32007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:33.298700 32008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "664dae6d3d0943e19634717ec4e72d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } } }
I20260812 00:12:33.298846 32008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:33.299237 32009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 664dae6d3d0943e19634717ec4e72d2e. Latest consensus state: current_term: 2 leader_uuid: "664dae6d3d0943e19634717ec4e72d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } } }
I20260812 00:12:33.299337 32009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:33.299232 32015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:33.300268 32015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:33.300534 32015 catalog_manager.cc:1259] Loaded cluster ID: a7639f61e0fb44e49e12b9ea008ab71d
I20260812 00:12:33.300721 32015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:12:33.301085 32015 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 00:12:33.306483 32015 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 00:12:33.309454 32015 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:33.309729 32015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Loaded TSK: 0
I20260812 00:12:33.309952 32015 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:12:33.375823 32005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:33.376093 32005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:33.376122 32005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:33.377491 32005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:33.377561 32005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:33.377600 32005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:12:33.379503 32005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:45483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=32789
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:33.379822 32005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:33.380162 32005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:33.386209 32035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:33.386237 32032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:33.386651 32033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:33.386560 32005 server_base.cc:1034] running on GCE node
I20260812 00:12:33.387075 32005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:33.387421 32005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:33.388666 32005 hybrid_clock.cc:648] HybridClock initialized: now 1786493553388639 us; error 48 us; skew 500 ppm
Aug 12 00:12:33 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:33.392027 32005 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:12:33.392855 32005 webserver.cc:460] Webserver started at http://127.16.99.193:32789/ using document root <none> and password file <none>
I20260812 00:12:33.393147 32005 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:33.393254 32005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:33.395031 32005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:12:33.395920 32042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.396198 32005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:33.396400 32005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "1b796088d1774259abebdbb7ce7bbab5"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "7584c7a8e4bebef678f31b9fc6f021d6"
server_key_iv: "32317a90de55c9391ba33cb9023f09f9"
server_key_version: "encryptionkey@0"
I20260812 00:12:33.396847 32005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:33.408021 32005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:33.410948 32005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:33.411162 32005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:33.411474 32005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:33.412055 32049 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:12:33.414208 32005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:12:33.414353 32005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 00:12:33.414461 32005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:12:33.415293 32005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:12:33.415366 32005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 00:12:33.415452 32049 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap starting.
I20260812 00:12:33.424461 32005 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:45483
I20260812 00:12:33.424683 32164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:45483 every 8 connection(s)
I20260812 00:12:33.424998 32005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:33.428360 32049 log.cc:824] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:33.431582 32049 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:33.431586 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 32005
I20260812 00:12:33.431993 32049 tablet_bootstrap.cc:492] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap complete.
I20260812 00:12:33.432035 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:37301
--local_ip_for_outbound_sockets=127.16.99.194
--tserver_master_addrs=127.16.99.254:42197
--webserver_port=46777
--webserver_interface=127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:33.433449 32049 ts_tablet_manager.cc:1397] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.010s
I20260812 00:12:33.435537 32049 raft_consensus.cc:348] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.435895 32049 raft_consensus.cc:729] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b796088d1774259abebdbb7ce7bbab5, State: Initialized, Role: FOLLOWER
Aug 12 00:12:33 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:33.436683 32049 consensus_queue.cc:260] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.437067 32049 ts_tablet_manager.cc:1428] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260812 00:12:33.441953 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.426175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:43539 (local address 127.16.99.254:42197)
0812 00:12:33.426566 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.426579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.426623 (+ 44us) server_negotiation.cc:407] Connection header received
0812 00:12:33.427117 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.427122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.427177 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.427353 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:33.428150 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.429453 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.430761 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.431244 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.431415 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.437281 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:33.437287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:33.437287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:33.437300 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:33.439680 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.440207 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.440211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.440211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.440335 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.440767 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.440768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.440769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.441055 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:33.441162 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.441520 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.441668 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":113,"threads_started":1}
I20260812 00:12:33.441967 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.425795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:43539)
0812 00:12:33.426028 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.426081 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.426843 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.427395 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.427399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.427782 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:33.428004 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.428010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.429610 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.429616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.430636 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.430640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.434022 (+ 3382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.434024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.434251 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.435072 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:33.435085 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:33.437110 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:33.439806 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:33.439816 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:33.439818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:33.440093 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:33.440483 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:33.440484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:33.440484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:33.440681 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:33.441151 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:33.441159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:33.441325 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.441444 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.441668 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":141,"threads_started":1}
I20260812 00:12:33.443322 32165 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42197
I20260812 00:12:33.443480 32165 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:33.443711 32165 heartbeater.cc:511] Master 127.16.99.254:42197 requested a full tablet report, sending...
I20260812 00:12:33.444582 31963 ts_manager.cc:194] Registered new tserver with Master: 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193:45483)
I20260812 00:12:33.445855 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:43539
I20260812 00:12:33.446973 32165 heartbeater.cc:503] Master 127.16.99.254:42197 was elected leader, sending a full tablet report...
I20260812 00:12:33.454238 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.445317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45483 (local address 127.0.0.1:58716)
0812 00:12:33.450539 (+ 5222us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.450587 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.450801 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.451039 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.451042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.451045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:33.451139 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.451142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.452162 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.452164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.453101 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.453105 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.453849 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.453850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.453946 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.453960 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.454075 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.454147 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5174,"thread_start_us":130,"threads_started":1}
I20260812 00:12:33.454238 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.445650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.16.99.193:45483)
0812 00:12:33.446533 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.446539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.450775 (+ 4236us) server_negotiation.cc:407] Connection header received
0812 00:12:33.450902 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.450905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.450922 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.451019 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:33.451315 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.452047 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.453214 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.453749 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.453847 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.453913 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.454097 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.454147 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":88,"threads_started":1}
I20260812 00:12:33.455068 32096 tablet_service.cc:1514] Processing DeleteTablet for tablet f01145e1c2c34b9cb73126e330d94c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:33 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:58716
I20260812 00:12:33.455694 32176 tablet_replica.cc:330] stopping tablet replica
I20260812 00:12:33.455829 32176 raft_consensus.cc:2229] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:33.455909 32176 raft_consensus.cc:2258] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:33.456259 32176 ts_tablet_manager.cc:1905] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:33.458811 32176 ts_tablet_manager.cc:1918] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:33.458943 32176 log.cc:1196] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/f01145e1c2c34b9cb73126e330d94c72
I20260812 00:12:33.459359 32176 ts_tablet_manager.cc:1939] T f01145e1c2c34b9cb73126e330d94c72 P 1b796088d1774259abebdbb7ce7bbab5: Deleting consensus metadata
I20260812 00:12:33.460018 31946 catalog_manager.cc:4841] TS 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193:45483): tablet f01145e1c2c34b9cb73126e330d94c72 (table test-table [id=62432c1fa5c644c9868325f9c14007c5]) successfully deleted
W20260812 00:12:33.531054 32171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:33.531313 32171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:33.531342 32171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:33.532691 32171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:33.532748 32171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:33.532779 32171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:12:33.534361 32171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:37301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=46777
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:33.534574 32171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:33.534768 32171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:33.540258 32182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:33.540248 32184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:33.540493 32171 server_base.cc:1034] running on GCE node
W20260812 00:12:33.540248 32181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:33.540910 32171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:33.541209 32171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:33.542435 32171 hybrid_clock.cc:648] HybridClock initialized: now 1786493553542417 us; error 48 us; skew 500 ppm
Aug 12 00:12:33 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:33.545322 32171 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:12:33.545853 32171 webserver.cc:460] Webserver started at http://127.16.99.194:46777/ using document root <none> and password file <none>
I20260812 00:12:33.546098 32171 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:33.546190 32171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:33.547863 32171 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:33.548612 32192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.548864 32171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:33.548972 32171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "288059d2de25484ead025e1137e916fd"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "ea701dc3ea580ecf5b6455757e969dd1"
server_key_iv: "222a898547bfdbbc9299552cbd2268bd"
server_key_version: "encryptionkey@0"
I20260812 00:12:33.549352 32171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:33.576241 32171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:33.580081 32171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:33.580344 32171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:33.580771 32171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:33.581292 32171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:33.581379 32171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.581475 32171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:33.581557 32171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.591080 32171 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:37301
I20260812 00:12:33.591197 32311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:37301 every 8 connection(s)
I20260812 00:12:33.591753 32171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:33.601620 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 32171
I20260812 00:12:33.602180 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:34447
--local_ip_for_outbound_sockets=127.16.99.195
--tserver_master_addrs=127.16.99.254:42197
--webserver_port=44443
--webserver_interface=127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:12:33 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:33.610539 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.592958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:45097 (local address 127.16.99.254:42197)
0812 00:12:33.593069 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.593071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.593438 (+ 367us) server_negotiation.cc:407] Connection header received
0812 00:12:33.594288 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.594290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.594318 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.594430 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:33.596404 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.597129 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.599211 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.599550 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.599660 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.604176 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:33.604183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:33.604184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:33.604201 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:33.607572 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.608479 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.608480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.608481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.608588 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.609224 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.609225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.609225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.609417 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:33.609504 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.610193 (+ 689us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.610409 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 00:12:33.611080 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.592921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:45097)
0812 00:12:33.593196 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.593254 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.594030 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.594670 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.594678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.595831 (+ 1153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:33.596221 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.596230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.597401 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.597402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.599042 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.599045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.599693 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.599695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.600012 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.600976 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:33.600994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:33.603946 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:33.607756 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:33.607789 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:33.607793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:33.608328 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:33.608749 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:33.608770 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:33.608770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:33.609091 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:33.609525 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:33.609541 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:33.609895 (+ 354us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.610165 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.610614 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":147,"threads_started":1}
I20260812 00:12:33.612141 32312 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42197
I20260812 00:12:33.612425 32312 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:33.612763 32312 heartbeater.cc:511] Master 127.16.99.254:42197 requested a full tablet report, sending...
I20260812 00:12:33.613390 31963 ts_manager.cc:194] Registered new tserver with Master: 288059d2de25484ead025e1137e916fd (127.16.99.194:37301)
I20260812 00:12:33.614248 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:45097
W20260812 00:12:33.738624 32315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:33.739010 32315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:33.739089 32315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:33.741091 32315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:33.741215 32315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:33.741325 32315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:12:33.743765 32315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:34447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=44443
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:33.744239 32315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:33.744699 32315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:33.753047 32320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:33.753047 32324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:33.753089 32322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:33.753394 32315 server_base.cc:1034] running on GCE node
I20260812 00:12:33.753705 32315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:33.754050 32315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:33.755334 32315 hybrid_clock.cc:648] HybridClock initialized: now 1786493553755315 us; error 33 us; skew 500 ppm
Aug 12 00:12:33 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:33.759887 32315 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:12:33.760743 32315 webserver.cc:460] Webserver started at http://127.16.99.195:44443/ using document root <none> and password file <none>
I20260812 00:12:33.761129 32315 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:33.761282 32315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:33.764097 32315 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:33.765313 32331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.765679 32315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:12:33.765864 32315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "1f1fec17f2e446c98f6a39bc40437a8e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "5678630ae2e907205a244a5d3fdabef8"
server_key_iv: "beff7880171ce41c21e85767339ada17"
server_key_version: "encryptionkey@0"
I20260812 00:12:33.766559 32315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:33.800884 32315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:33.805228 32315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:33.805512 32315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:33.805953 32315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:33.806550 32315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:33.806619 32315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.806658 32315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:33.806715 32315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:33.817808 32315 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:34447
I20260812 00:12:33.817873 32448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:34447 every 8 connection(s)
I20260812 00:12:33.818559 32315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:33.825927 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 32315
Aug 12 00:12:33 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493553, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:33.837960 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.820027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40133 (local address 127.16.99.254:42197)
0812 00:12:33.820196 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.820199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.820843 (+ 644us) server_negotiation.cc:407] Connection header received
0812 00:12:33.821787 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.821789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.821809 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.821908 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:33.823979 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.824798 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.827204 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.827575 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.827773 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.831818 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:33.831820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:33.831820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:33.831836 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:33.834821 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.836088 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.836089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.836090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.836186 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.836749 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.836750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.836751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.836947 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:33.837098 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.837633 (+ 535us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.837787 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260812 00:12:33.838445 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.820140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:40133)
0812 00:12:33.820606 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.820697 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.821597 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.822023 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.822027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.823374 (+ 1347us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:33.823811 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.823824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.824943 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.824944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.827046 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.827058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.827773 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.827774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.828031 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.828955 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:33.828974 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:33.831565 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:33.835377 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:33.835401 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:33.835405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:33.835845 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:33.836333 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:33.836334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:33.836334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:33.836559 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:33.837089 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:33.837099 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:33.837333 (+ 234us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.837549 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.837886 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":208,"threads_started":1}
I20260812 00:12:33.839406 32449 heartbeater.cc:347] Connected to a master server at 127.16.99.254:42197
I20260812 00:12:33.839627 32449 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:33.839969 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.826786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.16.99.254:42197)
0812 00:12:33.827092 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.827095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.827359 (+ 264us) server_negotiation.cc:407] Connection header received
0812 00:12:33.827544 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.827545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.827572 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.827788 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:33.828887 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.829768 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.831291 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.831660 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.831833 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.834643 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:33.834646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:33.834646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:33.834665 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:33.837161 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.837828 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.837838 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.837839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.837939 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:33.838644 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:33.838645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:33.838645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:33.838851 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:33.839261 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.839586 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.839755 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":137,"threads_started":1}
I20260812 00:12:33.840096 32449 heartbeater.cc:511] Master 127.16.99.254:42197 requested a full tablet report, sending...
I20260812 00:12:33.840781 31963 ts_manager.cc:194] Registered new tserver with Master: 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447)
I20260812 00:12:33.841351 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:33.841645 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:40133
I20260812 00:12:33.846684 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.841960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.16.99.254:42197)
0812 00:12:33.842034 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.842036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.842395 (+ 359us) server_negotiation.cc:407] Connection header received
0812 00:12:33.842532 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.842533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.842608 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.842730 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:33.843040 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.843809 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.845145 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.845504 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.845659 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.846003 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:33.846179 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:33.846406 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.846486 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.846526 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:12:33.847164 31963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:33.849789 31963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:33.854334 32096 tablet_service.cc:1467] Processing CreateTablet for tablet 66fb8e3eb79245568bbe7a3b8db4feec (DEFAULT_TABLE table=test-table [id=ebd6db02f67743a4b681ed9dcb3ea407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:33.854739 32096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fb8e3eb79245568bbe7a3b8db4feec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:33.856856 32049 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap starting.
I20260812 00:12:33.857450 32049 tablet_bootstrap.cc:654] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:33.858722 32049 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: No bootstrap required, opened a new log
I20260812 00:12:33.858834 32049 ts_tablet_manager.cc:1397] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:12:33.859079 32049 raft_consensus.cc:348] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.859246 32049 raft_consensus.cc:374] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:33.859375 32049 raft_consensus.cc:729] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b796088d1774259abebdbb7ce7bbab5, State: Initialized, Role: FOLLOWER
I20260812 00:12:33.859496 32049 consensus_queue.cc:260] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.859714 32049 ts_tablet_manager.cc:1428] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:33.862143 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.853883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:34447 (local address 127.0.0.1:44862)
0812 00:12:33.854008 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.854044 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.854128 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.855082 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.855082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.855093 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:33.855222 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.855226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.858052 (+ 2826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.858053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.859531 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.859564 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.861607 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.861615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.861721 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.861726 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.861834 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.861937 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":43}
I20260812 00:12:33.863178 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.853883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37301 (local address 127.0.0.1:34224)
0812 00:12:33.854422 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.854436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.854528 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.855009 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.855010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.855014 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:33.855161 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.855173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.857882 (+ 2709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.857884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.860076 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.860079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.862468 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.862469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.862659 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.862665 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.862778 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.862883 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":46,"thread_start_us":312,"threads_started":1}
I20260812 00:12:33.863314 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.854290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.16.99.195:34447)
0812 00:12:33.854712 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.854718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.854744 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:12:33.854809 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.854813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.854868 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.855157 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:33.855369 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.857842 (+ 2473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.859990 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.861409 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.862604 (+ 1195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.862847 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.862901 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.863004 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":204,"threads_started":1}
I20260812 00:12:33.864157 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.854199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.16.99.194:37301)
0812 00:12:33.854700 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.854716 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.854737 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:12:33.854811 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.854815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.854847 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.855021 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:33.855370 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.857598 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.860332 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.861915 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.862162 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.862395 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.863750 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.863874 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":117,"threads_started":1}
I20260812 00:12:33.865067 32381 tablet_service.cc:1467] Processing CreateTablet for tablet 66fb8e3eb79245568bbe7a3b8db4feec (DEFAULT_TABLE table=test-table [id=ebd6db02f67743a4b681ed9dcb3ea407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:33.865332 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 66fb8e3eb79245568bbe7a3b8db4feec (DEFAULT_TABLE table=test-table [id=ebd6db02f67743a4b681ed9dcb3ea407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:33.865697 32381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fb8e3eb79245568bbe7a3b8db4feec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:33.865808 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66fb8e3eb79245568bbe7a3b8db4feec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:33.871374 32463 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Bootstrap starting.
I20260812 00:12:33.871752 32464 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Bootstrap starting.
I20260812 00:12:33.872526 32463 tablet_bootstrap.cc:654] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:33.872632 32464 tablet_bootstrap.cc:654] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:33.873215 32464 log.cc:824] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:33.873260 32463 log.cc:824] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:33.874493 32464 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: No bootstrap required, opened a new log
I20260812 00:12:33.874581 32464 ts_tablet_manager.cc:1397] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260812 00:12:33.874926 32463 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: No bootstrap required, opened a new log
I20260812 00:12:33.874998 32463 ts_tablet_manager.cc:1397] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 00:12:33.877209 32463 raft_consensus.cc:348] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.877143 32464 raft_consensus.cc:348] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.877370 32463 raft_consensus.cc:374] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:33.877370 32464 raft_consensus.cc:374] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:33.877408 32464 raft_consensus.cc:729] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 288059d2de25484ead025e1137e916fd, State: Initialized, Role: FOLLOWER
I20260812 00:12:33.877408 32463 raft_consensus.cc:729] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1fec17f2e446c98f6a39bc40437a8e, State: Initialized, Role: FOLLOWER
I20260812 00:12:33.877542 32463 consensus_queue.cc:260] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.877542 32464 consensus_queue.cc:260] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.877890 32464 ts_tablet_manager.cc:1428] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:12:33.877890 32463 ts_tablet_manager.cc:1428] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:33.877959 32312 heartbeater.cc:503] Master 127.16.99.254:42197 was elected leader, sending a full tablet report...
I20260812 00:12:33.878027 32449 heartbeater.cc:503] Master 127.16.99.254:42197 was elected leader, sending a full tablet report...
I20260812 00:12:33.885066 32468 raft_consensus.cc:482] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:33.886019 32468 raft_consensus.cc:504] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.887588 32468 leader_election.cc:288] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 288059d2de25484ead025e1137e916fd (127.16.99.194:37301), 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193:45483)
I20260812 00:12:33.892455 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.887727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45483 (local address 127.16.99.195:43613)
0812 00:12:33.887810 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.887830 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.887900 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.888159 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.888165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.888168 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:33.888274 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.888282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.889432 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.889441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.890637 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.890645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.891949 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.891950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.892116 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.892121 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.892230 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.892317 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 00:12:33.892474 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.887855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:43613 (local address 127.16.99.193:45483)
0812 00:12:33.887951 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.887953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.887974 (+ 21us) server_negotiation.cc:407] Connection header received
0812 00:12:33.888015 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.888016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.888030 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.888134 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:33.888441 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.889253 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.890781 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.891784 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.891952 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.891975 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.892295 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.892377 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:12:33.894062 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.887900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37301 (local address 127.16.99.195:34847)
0812 00:12:33.888154 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.888174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:33.888233 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:33.888633 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:33.888634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:33.888637 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:33.888725 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.888727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.889726 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.889731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.891740 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:33.891744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.893243 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.893245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:33.893392 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.893399 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 00:12:33.893689 (+ 290us) client_negotiation.cc:240] Negotiation successful
0812 00:12:33.893729 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":159,"threads_started":1}
I20260812 00:12:33.893790 32117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fb8e3eb79245568bbe7a3b8db4feec" candidate_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b796088d1774259abebdbb7ce7bbab5" is_pre_election: true
I20260812 00:12:33.894282 32117 raft_consensus.cc:2454] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1fec17f2e446c98f6a39bc40437a8e in term 0.
I20260812 00:12:33.895008 32332 leader_election.cc:302] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5, 1f1fec17f2e446c98f6a39bc40437a8e; no voters:
I20260812 00:12:33.895921 32468 raft_consensus.cc:2781] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:33.896449 32468 raft_consensus.cc:482] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:33.896940 32468 raft_consensus.cc:3037] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:33.894423 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.888409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:34847 (local address 127.16.99.194:37301)
0812 00:12:33.888484 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.888486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.888493 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:12:33.888522 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.888523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.888533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.888584 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:33.888850 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.889562 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.891947 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.893071 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.893189 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.893198 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.893940 (+ 742us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.894242 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:12:33.904070 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fb8e3eb79245568bbe7a3b8db4feec" candidate_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288059d2de25484ead025e1137e916fd" is_pre_election: true
I20260812 00:12:33.905238 32263 raft_consensus.cc:2454] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1fec17f2e446c98f6a39bc40437a8e in term 0.
I20260812 00:12:33.905579 32468 raft_consensus.cc:504] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.906193 32468 leader_election.cc:288] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [CANDIDATE]: Term 1 election: Requested vote from peers 288059d2de25484ead025e1137e916fd (127.16.99.194:37301), 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193:45483)
I20260812 00:12:33.906615 32117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fb8e3eb79245568bbe7a3b8db4feec" candidate_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b796088d1774259abebdbb7ce7bbab5"
I20260812 00:12:33.906760 32117 raft_consensus.cc:3037] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:33.910295 32117 raft_consensus.cc:2454] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1fec17f2e446c98f6a39bc40437a8e in term 1.
I20260812 00:12:33.911208 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fb8e3eb79245568bbe7a3b8db4feec" candidate_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "288059d2de25484ead025e1137e916fd"
I20260812 00:12:33.911551 32332 leader_election.cc:302] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5, 1f1fec17f2e446c98f6a39bc40437a8e; no voters:
I20260812 00:12:33.911664 32263 raft_consensus.cc:3037] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:33.912196 32468 raft_consensus.cc:2781] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:33.912600 32468 raft_consensus.cc:686] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 LEADER]: Becoming Leader. State: Replica: 1f1fec17f2e446c98f6a39bc40437a8e, State: Running, Role: LEADER
I20260812 00:12:33.912899 32468 consensus_queue.cc:237] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:33.914548 32263 raft_consensus.cc:2454] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1fec17f2e446c98f6a39bc40437a8e in term 1.
I20260812 00:12:33.914592 31962 catalog_manager.cc:5495] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195). New cstate: current_term: 1 leader_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } health_report { overall_health: UNKNOWN } } }
I20260812 00:12:33.924090 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.919964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.16.99.195:34447)
0812 00:12:33.920057 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:33.920059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:33.920070 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:12:33.920122 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:33.920123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:33.920165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:33.920429 (+ 264us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:33.920694 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.921326 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.922549 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:33.922935 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:33.923122 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:33.923403 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:33.923596 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:33.923799 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:33.923842 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 00:12:33.923878 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 00:12:33.926146 32166 tablet.cc:2369] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:33.927609 32117 raft_consensus.cc:1261] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Refusing update from remote peer 1f1fec17f2e446c98f6a39bc40437a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:33.927709 32263 raft_consensus.cc:1261] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Refusing update from remote peer 1f1fec17f2e446c98f6a39bc40437a8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:33.928135 32468 consensus_queue.cc:1035] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:33.928377 32470 consensus_queue.cc:1035] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:33.933269 32473 mvcc.cc:205] Tried to move back new op lower bound from 7317477596883562496 to 7317477596829384704. Current Snapshot: MvccSnapshot[applied={T|T < 7317477596883562496 or (T in {7317477596883562496})}]
I20260812 00:12:33.936424 31963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:33.936620 31963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:33.937970 31963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Sending DeleteTablet for 3 replicas of tablet 66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:33.938589 32381 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fb8e3eb79245568bbe7a3b8db4feec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:33 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:44862
I20260812 00:12:33.938566 32096 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fb8e3eb79245568bbe7a3b8db4feec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:33 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:58716
I20260812 00:12:33.938630 32243 tablet_service.cc:1514] Processing DeleteTablet for tablet 66fb8e3eb79245568bbe7a3b8db4feec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:33 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34224
I20260812 00:12:33.938820 32176 tablet_replica.cc:330] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: stopping tablet replica
I20260812 00:12:33.938930 32176 raft_consensus.cc:2229] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:33.939045 32176 raft_consensus.cc:2258] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:33.939147 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 32005
I20260812 00:12:33.939306 32485 tablet_replica.cc:330] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: stopping tablet replica
I20260812 00:12:33.939620 32485 raft_consensus.cc:2229] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:33.939919 32485 raft_consensus.cc:2258] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:33.941875 32485 ts_tablet_manager.cc:1905] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:33.946033 32485 ts_tablet_manager.cc:1918] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:33.946389 32485 log.cc:1196] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:33.947229 32485 ts_tablet_manager.cc:1939] T 66fb8e3eb79245568bbe7a3b8db4feec P 288059d2de25484ead025e1137e916fd: Deleting consensus metadata
I20260812 00:12:33.948403 31949 catalog_manager.cc:4841] TS 288059d2de25484ead025e1137e916fd (127.16.99.194:37301): tablet 66fb8e3eb79245568bbe7a3b8db4feec (table test-table [id=ebd6db02f67743a4b681ed9dcb3ea407]) successfully deleted
I20260812 00:12:33.950539 32484 tablet_replica.cc:330] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: stopping tablet replica
I20260812 00:12:33.950807 32484 raft_consensus.cc:2229] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:33.951032 32484 raft_consensus.cc:2258] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:33.952135 32484 ts_tablet_manager.cc:1905] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:33.955098 32484 ts_tablet_manager.cc:1918] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:33.955188 32484 log.cc:1196] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:33.955667 32484 ts_tablet_manager.cc:1939] T 66fb8e3eb79245568bbe7a3b8db4feec P 1f1fec17f2e446c98f6a39bc40437a8e: Deleting consensus metadata
I20260812 00:12:33.956430 31947 catalog_manager.cc:4841] TS 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447): tablet 66fb8e3eb79245568bbe7a3b8db4feec (table test-table [id=ebd6db02f67743a4b681ed9dcb3ea407]) successfully deleted
W20260812 00:12:33.957371 31946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:45483 (error 108)
I20260812 00:12:33.958123 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 32171
I20260812 00:12:33.958173 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:33.958009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45483 (local address 127.0.0.1:58722)
0812 00:12:33.958070 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:33.958134 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 00:12:33.966668 31946 catalog_manager.cc:4584] TS 1b796088d1774259abebdbb7ce7bbab5 (127.16.99.193:45483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66fb8e3eb79245568bbe7a3b8db4feec: Network error: Client connection negotiation failed: client connection to 127.16.99.193:45483: connect: Connection refused (error 111)
I20260812 00:12:33.972074 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 32315
I20260812 00:12:33.981765 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 31929
I20260812 00:12:33.993188 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42197
--webserver_interface=127.16.99.254
--webserver_port=32927
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:34.126700 32487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:34.127084 32487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:34.127128 32487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:34.128990 32487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:34.129074 32487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:34.129101 32487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:34.129124 32487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:34.129141 32487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:34.131543 32487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:42197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:34.132025 32487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:34.132579 32487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:34.140866 32492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.140890 32493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.140869 32495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:34.141422 32487 server_base.cc:1034] running on GCE node
I20260812 00:12:34.141909 32487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:34.142582 32487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:34.143818 32487 hybrid_clock.cc:648] HybridClock initialized: now 1786493554143768 us; error 70 us; skew 500 ppm
Aug 12 00:12:34 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:34.148378 32487 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:34.149169 32487 webserver.cc:460] Webserver started at http://127.16.99.254:32927/ using document root <none> and password file <none>
I20260812 00:12:34.149484 32487 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:34.149580 32487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:34.151922 32487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:34.153491 32503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.153877 32487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:34.154038 32487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "664dae6d3d0943e19634717ec4e72d2e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "6c2701a08dc1724eb9a2e8fc1ee97229"
server_key_iv: "ec909bef0be98c2dfe7668c445b09b58"
server_key_version: "encryptionkey@0"
I20260812 00:12:34.154716 32487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:34.172435 32487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:34.176394 32487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:34.176862 32487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:34.185396 32487 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:42197
I20260812 00:12:34.185514 32556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:42197 every 8 connection(s)
I20260812 00:12:34.186213 32487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:34.190016 32557 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:12:34.191428 32557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Bootstrap starting.
I20260812 00:12:34.194023 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 32487
I20260812 00:12:34.194852 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:45483
--local_ip_for_outbound_sockets=127.16.99.193
--tserver_master_addrs=127.16.99.254:42197
--webserver_port=32789
--webserver_interface=127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:34.208695 32557 log.cc:824] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:34.214928 32557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:34.215682 32557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e: Bootstrap complete.
I20260812 00:12:34.220376 32557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:34.220821 32557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664dae6d3d0943e19634717ec4e72d2e, State: Initialized, Role: FOLLOWER
I20260812 00:12:34.221172 32557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:34.221273 32557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:34.221355 32557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:34.221472 32557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:12:34.224316 32557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:34.224598 32557 leader_election.cc:302] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 664dae6d3d0943e19634717ec4e72d2e; no voters:
I20260812 00:12:34.225064 32557 leader_election.cc:288] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 00:12:34.225136 32562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 3 FOLLOWER]: Leader election won for term 3
I20260812 00:12:34.225380 32562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [term 3 LEADER]: Becoming Leader. State: Replica: 664dae6d3d0943e19634717ec4e72d2e, State: Running, Role: LEADER
I20260812 00:12:34.225596 32562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } }
I20260812 00:12:34.225680 32557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:34.226368 32565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 664dae6d3d0943e19634717ec4e72d2e. Latest consensus state: current_term: 3 leader_uuid: "664dae6d3d0943e19634717ec4e72d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } } }
I20260812 00:12:34.226430 32564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "664dae6d3d0943e19634717ec4e72d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "664dae6d3d0943e19634717ec4e72d2e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 42197 } } }
I20260812 00:12:34.226552 32565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:34.226624 32564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 664dae6d3d0943e19634717ec4e72d2e [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:34.228051 32567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:34.229673 32567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:34.229949 32567 catalog_manager.cc:1259] Loaded cluster ID: a7639f61e0fb44e49e12b9ea008ab71d
I20260812 00:12:34.230093 32567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 00:12:34.231026 32567 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2a999a99ad google::LogMessage::Fail() at ??:0
@ 0x7f2a999ac88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2a999a94de google::LogMessage::Flush() at ??:0
@ 0x7f2a999aaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2aa18ce0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2aa18d78b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2a9b8770d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2a9b86c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2a9c6d06db start_thread at ??:0
@ 0x7f2a984a761f clone at ??:0
W20260812 00:12:34.469394 32560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:34.470301 32560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:34.470873 32560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:34.472676 32560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:34.472780 32560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:34.472837 32560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:12:34.475247 32560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:45483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=32789
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:34.475684 32560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:34.475975 32560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:34.482841 32596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.482810 32598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.482841 32595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:34.483321 32560 server_base.cc:1034] running on GCE node
I20260812 00:12:34.483738 32560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:34.484098 32560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:34.485356 32560 hybrid_clock.cc:648] HybridClock initialized: now 1786493554485313 us; error 60 us; skew 500 ppm
Aug 12 00:12:34 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:34.488924 32560 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:12:34.489534 32560 webserver.cc:460] Webserver started at http://127.16.99.193:32789/ using document root <none> and password file <none>
I20260812 00:12:34.489850 32560 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:34.489905 32560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:34.491746 32560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:34.492717 32607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.493054 32560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:34.493172 32560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "1b796088d1774259abebdbb7ce7bbab5"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "7584c7a8e4bebef678f31b9fc6f021d6"
server_key_iv: "32317a90de55c9391ba33cb9023f09f9"
server_key_version: "encryptionkey@0"
I20260812 00:12:34.493742 32560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:34.517217 32560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:34.523309 32560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:34.523744 32560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:34.524811 32560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:34.528116 32614 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:12:34.539749 32560 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:12:34.540521 32560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.001s
I20260812 00:12:34.540858 32560 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:12:34.544303 32560 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:12:34.544462 32560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.003s
I20260812 00:12:34.544659 32614 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap starting.
I20260812 00:12:34.554160 32560 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:45483
I20260812 00:12:34.554260 32728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:45483 every 8 connection(s)
I20260812 00:12:34.554805 32560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:12:34.559406 32614 log.cc:824] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:34.562696 32614 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:12:34.563198 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 32560
I20260812 00:12:34.563303 32614 tablet_bootstrap.cc:492] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Bootstrap complete.
I20260812 00:12:34.563764 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:37301
--local_ip_for_outbound_sockets=127.16.99.194
--tserver_master_addrs=127.16.99.254:42197
--webserver_port=46777
--webserver_interface=127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:12:34.565531 32614 ts_tablet_manager.cc:1397] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.011s
I20260812 00:12:34.568379 32614 raft_consensus.cc:348] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:34.568946 32614 raft_consensus.cc:729] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b796088d1774259abebdbb7ce7bbab5, State: Initialized, Role: FOLLOWER
I20260812 00:12:34.569253 32614 consensus_queue.cc:260] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:34.569792 32614 ts_tablet_manager.cc:1428] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5: Time spent starting tablet: real 0.004s user 0.002s sys 0.005s
W20260812 00:12:34.682210 32732 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:12:34.555704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:53853)
0812 00:12:34.555940 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.555984 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:34.556612 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:34.681885 (+125273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":118,"threads_started":1}
W20260812 00:12:34.682652 32730 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:42197 (0 consecutive failures): Network error: Failed to ping master at 127.16.99.254:42197: Client connection negotiation failed: client connection to 127.16.99.254:42197: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:12:34.683166 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.683009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:40691)
0812 00:12:34.683076 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.683136 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 00:12:34.683678 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.683520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:35535)
0812 00:12:34.683586 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.683639 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 00:12:34.683887 32730 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:12:34.705667 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:34.706022 32734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:34.706068 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:34.708043 32734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:34.708134 32734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:34.708181 32734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:12:34.710527 32734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:37301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=46777
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:34.710932 32734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:34.711314 32734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:34.719193 32740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:34.719439 32734 server_base.cc:1034] running on GCE node
W20260812 00:12:34.719244 32741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.719213 32743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:34.719969 32734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:34.720381 32734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:34.721619 32734 hybrid_clock.cc:648] HybridClock initialized: now 1786493554721575 us; error 83 us; skew 500 ppm
Aug 12 00:12:34 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:34.725811 32734 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:12:34.726876 32734 webserver.cc:460] Webserver started at http://127.16.99.194:46777/ using document root <none> and password file <none>
I20260812 00:12:34.727370 32734 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:34.727515 32734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:34.730067 32734 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:12:34.731608 32750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.731901 32734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:34.732096 32734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "288059d2de25484ead025e1137e916fd"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "ea701dc3ea580ecf5b6455757e969dd1"
server_key_iv: "222a898547bfdbbc9299552cbd2268bd"
server_key_version: "encryptionkey@0"
I20260812 00:12:34.732839 32734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:34.754237 32734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:34.758138 32734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:34.758505 32734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:34.758976 32734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:34.759580 32734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:34.759698 32734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.759764 32734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:34.759821 32734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.772894 32734 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:37301
I20260812 00:12:34.772964 404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:37301 every 8 connection(s)
I20260812 00:12:34.773666 32734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:34.775815 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.774912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:40131)
0812 00:12:34.775262 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.775461 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":174,"threads_started":1}
W20260812 00:12:34.776862 405 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:42197 (0 consecutive failures): Network error: Failed to ping master at 127.16.99.254:42197: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
I20260812 00:12:34.777431 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.777275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:41479)
0812 00:12:34.777352 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.777391 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 00:12:34.777895 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.777781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:60995)
0812 00:12:34.777836 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.777870 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 00:12:34.778209 405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:12:34.778827 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 32734
I20260812 00:12:34.779414 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:34447
--local_ip_for_outbound_sockets=127.16.99.195
--tserver_master_addrs=127.16.99.254:42197
--webserver_port=44443
--webserver_interface=127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:34.926275 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:34.926712 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:34.926795 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:34.928663 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:34.928795 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:34.928870 408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:12:34.931357 408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:34447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=44443
--tserver_master_addrs=127.16.99.254:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:34.931829 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:34.932245 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:34.939564 414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.939561 415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:34.939561 417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:34.940157 408 server_base.cc:1034] running on GCE node
I20260812 00:12:34.940461 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:34.940819 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:34.942034 408 hybrid_clock.cc:648] HybridClock initialized: now 1786493554941997 us; error 48 us; skew 500 ppm
Aug 12 00:12:34 dist-test-slave-zpfg krb5kdc[31347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:34.946041 408 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:12:34.946893 408 webserver.cc:460] Webserver started at http://127.16.99.195:44443/ using document root <none> and password file <none>
I20260812 00:12:34.947224 408 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:34.947314 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:34.949565 408 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:12:34.950680 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.950984 408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:34.951134 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "1f1fec17f2e446c98f6a39bc40437a8e"
format_stamp: "Formatted at 2026-08-12 00:12:32 on dist-test-slave-zpfg"
server_key: "5678630ae2e907205a244a5d3fdabef8"
server_key_iv: "beff7880171ce41c21e85767339ada17"
server_key_version: "encryptionkey@0"
I20260812 00:12:34.951719 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:34.974615 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:34.978470 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:34.978734 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:34.979121 408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:34.979591 408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:34.979674 408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.979763 408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:34.979801 408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:34.989441 408 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:34447
I20260812 00:12:34.989521 544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:34447 every 8 connection(s)
I20260812 00:12:34.990144 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:34.991751 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.991258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:49945)
0812 00:12:34.991484 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.991597 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260812 00:12:34.992141 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 408
W20260812 00:12:34.992653 545 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:42197 (0 consecutive failures): Network error: Failed to ping master at 127.16.99.254:42197: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
I20260812 00:12:34.993233 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.992950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:53965)
0812 00:12:34.993022 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.993200 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 00:12:34.993882 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:34.993689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:39437)
0812 00:12:34.993804 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:34.993843 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260812 00:12:34.994150 545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:12:35.684727 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.684393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:56647)
0812 00:12:35.684598 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.684669 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":127,"threads_started":1}
I20260812 00:12:35.779062 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.778756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:41753)
0812 00:12:35.778916 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.778974 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260812 00:12:35.839190 551 raft_consensus.cc:482] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:35.839350 551 raft_consensus.cc:504] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:35.839792 551 leader_election.cc:288] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 288059d2de25484ead025e1137e916fd (127.16.99.194:37301), 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447)
Aug 12 00:12:35 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.194@KRBTEST.COM
Aug 12 00:12:35 dist-test-slave-zpfg krb5kdc[31347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493554, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.195@KRBTEST.COM
I20260812 00:12:35.852262 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.840271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:37321 (local address 127.16.99.194:37301)
0812 00:12:35.840580 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.840589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.840629 (+ 40us) server_negotiation.cc:407] Connection header received
0812 00:12:35.841114 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.841116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.841182 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.841360 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.842218 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.843303 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.844603 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.845093 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.845272 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.848091 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.848097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.848097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:35.848110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.850212 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:35.850873 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:35.850877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:35.850877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:35.851006 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:35.851297 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:35.851298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:35.851298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:35.851574 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.851683 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.852003 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.852142 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":72,"threads_started":1}
I20260812 00:12:35.852257 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.840011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:34447 (local address 127.16.99.193:43391)
0812 00:12:35.840076 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.840090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:35.840169 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:35.841341 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:35.841345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:35.841785 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:35.842065 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.842071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.843507 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.843508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.844440 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.844443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.845212 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.845213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.845354 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.846034 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:35.846042 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:35.848269 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:35.850508 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:35.850509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:35.850509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:35.850802 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:35.851108 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:35.851109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:35.851109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:35.851225 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:35.851655 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:35.851663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:35.851829 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 00:12:35.851946 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:12:35.852163 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":30}
I20260812 00:12:35.852386 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.840015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37301 (local address 127.16.99.193:37321)
0812 00:12:35.840276 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.840286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:35.840334 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:35.841366 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:35.841368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:35.841786 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:35.842064 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.842071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.843441 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.843443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.844461 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:35.844464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.845222 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.845224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:35.845353 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.846073 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:35.846078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:35.847871 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:35.850327 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:35.850368 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:35.850371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:35.850749 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:35.851089 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:35.851090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:35.851091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:35.851222 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:35.851657 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:35.851663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:35.851829 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 00:12:35.851946 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:12:35.852239 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":253,"thread_start_us":82,"threads_started":1}
I20260812 00:12:35.852288 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.840171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:43391 (local address 127.16.99.195:34447)
0812 00:12:35.840353 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:35.840367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:35.840395 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:12:35.840956 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:35.840969 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:35.841040 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:35.841270 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:35.842220 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.843348 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.844552 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:35.845091 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:35.845257 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:35.848406 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:35.848419 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:35.848419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:35.848431 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:35.850417 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:35.850902 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:35.850905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:35.850905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:35.851007 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:35.851328 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:35.851329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:35.851329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:35.851575 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:35.851677 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:35.852004 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 00:12:35.852171 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
W20260812 00:12:35.853327 32611 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 288059d2de25484ead025e1137e916fd (127.16.99.194:37301): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
W20260812 00:12:35.853422 32609 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:35.853482 32609 leader_election.cc:302] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5; no voters: 1f1fec17f2e446c98f6a39bc40437a8e, 288059d2de25484ead025e1137e916fd
I20260812 00:12:35.853613 551 raft_consensus.cc:2726] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:12:35.994930 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:35.994623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:49187)
0812 00:12:35.994810 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:35.994879 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260812 00:12:36.685638 556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:36.685359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:34237)
0812 00:12:36.685531 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:36.685584 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260812 00:12:36.779982 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:36.779624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:51757)
0812 00:12:36.779857 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:36.779927 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":142,"threads_started":1}
I20260812 00:12:36.995900 558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:36.995525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:37743)
0812 00:12:36.995755 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:36.995850 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":139,"threads_started":1}
I20260812 00:12:37.465806 559 raft_consensus.cc:482] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:37.465919 559 raft_consensus.cc:504] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:37.466099 559 leader_election.cc:288] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 288059d2de25484ead025e1137e916fd (127.16.99.194:37301), 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447)
W20260812 00:12:37.466553 32609 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
W20260812 00:12:37.466688 32611 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 288059d2de25484ead025e1137e916fd (127.16.99.194:37301): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:37.466707 32611 leader_election.cc:302] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5; no voters: 1f1fec17f2e446c98f6a39bc40437a8e, 288059d2de25484ead025e1137e916fd
I20260812 00:12:37.466794 559 raft_consensus.cc:2726] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:12:37.686599 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:37.686296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:54277)
0812 00:12:37.686499 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:37.686549 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":135,"threads_started":1}
I20260812 00:12:37.780855 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:37.780507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:55231)
0812 00:12:37.780664 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:37.780716 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260812 00:12:37.997066 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:37.996755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:33933)
0812 00:12:37.996937 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:37.996991 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260812 00:12:38.687446 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.687170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:50495)
0812 00:12:38.687346 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.687395 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260812 00:12:38.781920 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.781597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:42807)
0812 00:12:38.781795 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.781852 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":113,"threads_started":1}
I20260812 00:12:38.998208 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:38.997796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:58651)
0812 00:12:38.998098 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:38.998144 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":222,"threads_started":1}
I20260812 00:12:39.688599 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:39.688349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:43355)
0812 00:12:39.688499 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:39.688544 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260812 00:12:39.782912 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:39.782626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:55477)
0812 00:12:39.782783 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:39.782846 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260812 00:12:39.999058 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:39.998822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:33921)
0812 00:12:39.998959 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:39.999003 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260812 00:12:40.689415 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:40.689154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:41191)
0812 00:12:40.689325 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:40.689368 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260812 00:12:40.700138 570 raft_consensus.cc:482] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:40.700299 570 raft_consensus.cc:504] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:40.700559 570 leader_election.cc:288] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 288059d2de25484ead025e1137e916fd (127.16.99.194:37301), 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447)
W20260812 00:12:40.700986 32609 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
W20260812 00:12:40.701066 32611 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 288059d2de25484ead025e1137e916fd (127.16.99.194:37301): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:40.701148 32611 leader_election.cc:302] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5; no voters: 1f1fec17f2e446c98f6a39bc40437a8e, 288059d2de25484ead025e1137e916fd
I20260812 00:12:40.701238 570 raft_consensus.cc:2726] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:12:40.783756 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:40.783487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:58333)
0812 00:12:40.783661 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:40.783704 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260812 00:12:41.000171 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:40.999852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:53859)
0812 00:12:41.000035 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:41.000085 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260812 00:12:41.690248 573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:41.689944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:36583)
0812 00:12:41.690082 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:41.690127 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260812 00:12:41.784825 574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:41.784493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:38669)
0812 00:12:41.784701 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:41.784757 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
I20260812 00:12:42.001029 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:42.000765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:58583)
0812 00:12:42.000934 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:42.000979 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260812 00:12:42.691130 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:42.690852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:36043)
0812 00:12:42.691027 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:42.691082 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":68,"threads_started":1}
I20260812 00:12:42.785813 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:42.785556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:42511)
0812 00:12:42.785717 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:42.785762 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260812 00:12:43.001987 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:43.001719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:47883)
0812 00:12:43.001888 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:43.001936 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260812 00:12:43.431430 579 raft_consensus.cc:482] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:43.431553 579 raft_consensus.cc:504] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:43.431921 579 leader_election.cc:288] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 288059d2de25484ead025e1137e916fd (127.16.99.194:37301), 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447)
W20260812 00:12:43.432477 32611 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 288059d2de25484ead025e1137e916fd (127.16.99.194:37301): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
W20260812 00:12:43.432591 32609 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:43.432749 32609 leader_election.cc:302] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5; no voters: 1f1fec17f2e446c98f6a39bc40437a8e, 288059d2de25484ead025e1137e916fd
I20260812 00:12:43.432873 579 raft_consensus.cc:2726] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:12:43.692044 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:43.691798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:60333)
0812 00:12:43.691945 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:43.691995 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260812 00:12:43.786875 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:43.786584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:45673)
0812 00:12:43.786754 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:43.786805 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260812 00:12:44.003023 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:44.002699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:42879)
0812 00:12:44.002922 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:44.002972 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":127,"threads_started":1}
I20260812 00:12:44.693048 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:44.692755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:48209)
0812 00:12:44.692950 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:44.692998 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":122,"threads_started":1}
I20260812 00:12:44.787999 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:44.787747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:42887)
0812 00:12:44.787895 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:44.787946 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260812 00:12:45.004206 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:45.003872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:55301)
0812 00:12:45.004082 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:45.004140 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260812 00:12:45.693918 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:45.693643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:37023)
0812 00:12:45.693819 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:45.693868 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260812 00:12:45.788920 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:45.788652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:45243)
0812 00:12:45.788824 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:45.788868 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260812 00:12:46.005192 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:46.004922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:44313)
0812 00:12:46.005082 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:46.005140 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260812 00:12:46.694886 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:46.694635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:52469)
0812 00:12:46.694793 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:46.694839 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260812 00:12:46.789721 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:46.789485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:47105)
0812 00:12:46.789623 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:46.789671 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260812 00:12:47.006260 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:47.005972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:52627)
0812 00:12:47.006139 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:47.006191 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260812 00:12:47.695811 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:47.695463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:51731)
0812 00:12:47.695709 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:47.695763 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":75,"threads_started":1}
I20260812 00:12:47.790757 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:47.790442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:37543)
0812 00:12:47.790649 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:47.790696 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
I20260812 00:12:48.007339 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:48.006954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:44669)
0812 00:12:48.007235 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:48.007287 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":77,"threads_started":1}
I20260812 00:12:48.323879 595 raft_consensus.cc:482] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:48.323988 595 raft_consensus.cc:504] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "288059d2de25484ead025e1137e916fd" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37301 } } peers { permanent_uuid: "1f1fec17f2e446c98f6a39bc40437a8e" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 34447 } } peers { permanent_uuid: "1b796088d1774259abebdbb7ce7bbab5" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45483 } }
I20260812 00:12:48.324229 595 leader_election.cc:288] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 288059d2de25484ead025e1137e916fd (127.16.99.194:37301), 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447)
W20260812 00:12:48.324802 32609 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f1fec17f2e446c98f6a39bc40437a8e (127.16.99.195:34447): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
W20260812 00:12:48.324879 32611 leader_election.cc:341] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 288059d2de25484ead025e1137e916fd (127.16.99.194:37301): Not found: Tablet not found: 66fb8e3eb79245568bbe7a3b8db4feec
I20260812 00:12:48.324918 32611 leader_election.cc:302] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b796088d1774259abebdbb7ce7bbab5; no voters: 1f1fec17f2e446c98f6a39bc40437a8e, 288059d2de25484ead025e1137e916fd
I20260812 00:12:48.325035 595 raft_consensus.cc:2726] T 66fb8e3eb79245568bbe7a3b8db4feec P 1b796088d1774259abebdbb7ce7bbab5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:12:48.696731 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:48.696466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:37149)
0812 00:12:48.696634 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:48.696681 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260812 00:12:48.791949 598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:48.791669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:33305)
0812 00:12:48.791837 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:48.791896 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260812 00:12:49.008350 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:49.008055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:36883)
0812 00:12:49.008231 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:49.008296 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260812 00:12:49.697767 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:49.697524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.193:50163)
0812 00:12:49.697661 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:49.697721 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":126,"threads_started":1}
I20260812 00:12:49.792888 601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:49.792604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.194:52653)
0812 00:12:49.792772 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:49.792840 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260812 00:12:50.002988 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 32560
I20260812 00:12:50.009258 603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:50.008992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:42197 (local address 127.16.99.195:56511)
0812 00:12:50.009158 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:50.009200 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:42197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260812 00:12:50.018452 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 32734
I20260812 00:12:50.025192 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 408
2026-08-12T00:12:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19166 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:12:50 dist-test-slave-zpfg krb5kdc[606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:12:50 dist-test-slave-zpfg krb5kdc[606](info): set up 2 sockets
Aug 12 00:12:50 dist-test-slave-zpfg krb5kdc[606](info): commencing operation
krb5kdc: starting...
W20260812 00:12:51.186564 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:12:51 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:12:51.322690 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37271
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:51.411414 620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:51.411690 620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:51.411729 620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:51.412922 620 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 00:12:51.412992 620 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 00:12:51.413013 620 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 00:12:51.413033 620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:12:51.413064 620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:51.413092 620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:51.413120 620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:12:51.413148 620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:12:51.415086 620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:51.415419 620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:51.415673 620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:12:51.421741 620 server_base.cc:1034] running on GCE node
W20260812 00:12:51.421700 625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:51.421717 626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:51.421700 628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:51.422231 620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:51.422619 620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:51.423830 620 hybrid_clock.cc:648] HybridClock initialized: now 1786493571423792 us; error 56 us; skew 500 ppm
Aug 12 00:12:51 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:51.431944 620 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:12:51.432529 620 webserver.cc:460] Webserver started at http://127.16.99.254:33353/ using document root <none> and password file <none>
I20260812 00:12:51.432816 620 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:51.432868 620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:51.433049 620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:51.434319 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "57e08f90af1c4069a60f4c36a30ac556"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "f34cb93ff3e90c738a6aaef11dc7d4d3"
server_key_iv: "ae34b117b83f42189c7a03321cece8e8"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.434810 620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "57e08f90af1c4069a60f4c36a30ac556"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "f34cb93ff3e90c738a6aaef11dc7d4d3"
server_key_iv: "ae34b117b83f42189c7a03321cece8e8"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.436558 620 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:51.437669 635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:51.437930 620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:51.438047 620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "57e08f90af1c4069a60f4c36a30ac556"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "f34cb93ff3e90c738a6aaef11dc7d4d3"
server_key_iv: "ae34b117b83f42189c7a03321cece8e8"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.438115 620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:51.446538 620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:51.449180 620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:51.449317 620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:51.453888 620 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:37271
I20260812 00:12:51.453959 690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:37271 every 8 connection(s)
I20260812 00:12:51.454414 620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:12:51.455530 691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:51.458081 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Bootstrap starting.
I20260812 00:12:51.458863 691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:51.459198 691 log.cc:824] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:51.460027 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 620
I20260812 00:12:51.460175 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:12:51.460187 691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: No bootstrap required, opened a new log
I20260812 00:12:51.460450 16783 external_mini_cluster.cc:1428] Setting key d9669315d9c32659a04084db37edfef9
I20260812 00:12:51.462504 691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:12:51.462670 691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:51.462694 691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e08f90af1c4069a60f4c36a30ac556, State: Initialized, Role: FOLLOWER
I20260812 00:12:51.462867 691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:12:51.462980 691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:12:51.463020 691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:12:51.463083 691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:51.464262 691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:12:51.464370 691 leader_election.cc:302] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57e08f90af1c4069a60f4c36a30ac556; no voters:
I20260812 00:12:51.464556 691 leader_election.cc:288] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:12:51.464681 696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:51.464876 696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 LEADER]: Becoming Leader. State: Replica: 57e08f90af1c4069a60f4c36a30ac556, State: Running, Role: LEADER
I20260812 00:12:51.464892 691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:12:51.464967 696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:12:51.465335 696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57e08f90af1c4069a60f4c36a30ac556. Latest consensus state: current_term: 1 leader_uuid: "57e08f90af1c4069a60f4c36a30ac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } } }
I20260812 00:12:51.465454 696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:51.465348 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57e08f90af1c4069a60f4c36a30ac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } } }
I20260812 00:12:51.465632 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: This master's current role is: LEADER
I20260812 00:12:51.466502 704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:12:51.467188 704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:12:51.468942 704 catalog_manager.cc:1347] Generated new cluster ID: 3b39026ff976421eab4e3252ab4545ce
I20260812 00:12:51.469002 704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:12:51 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:51.475459 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:51.461085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.16.99.254:37271)
0812 00:12:51.461323 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:51.461332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:51.461359 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:12:51.462531 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:51.462560 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:51.462625 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:51.462829 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:51.463741 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.465425 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.466781 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.467449 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.467558 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:51.470847 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:51.470853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:51.470853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:51.470869 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:51.473287 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:51.473858 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:51.473861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:51.473862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:51.474059 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:51.474387 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:51.474388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:51.474388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:51.474744 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:51.474899 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:51.475069 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 00:12:51.475221 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260812 00:12:51.496909 704 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:12:51.497812 704 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:12:51.504854 704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Generated new TSK 0
I20260812 00:12:51.505113 704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:12:51.538492 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37271
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:51.625677 719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:51.625911 719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:51.625937 719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:51.627233 719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:51.627285 719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:51.627315 719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:12:51.628921 719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:51.629201 719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:51.629462 719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:51.634859 727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:51.634861 725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:51.634859 724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:51.635352 719 server_base.cc:1034] running on GCE node
I20260812 00:12:51.635653 719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:51.635911 719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:51.637094 719 hybrid_clock.cc:648] HybridClock initialized: now 1786493571637083 us; error 31 us; skew 500 ppm
Aug 12 00:12:51 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:51.639891 719 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:12:51.640353 719 webserver.cc:460] Webserver started at http://127.16.99.193:44291/ using document root <none> and password file <none>
I20260812 00:12:51.640605 719 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:51.640691 719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:51.640843 719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:51.641933 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "c1855aa087f44db2a1ec2d27cefc02d4"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "bae3d4881b05bec936873b99dd745301"
server_key_iv: "49dacd03d79e3082cbbcc6f338ab1340"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.642418 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "c1855aa087f44db2a1ec2d27cefc02d4"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "bae3d4881b05bec936873b99dd745301"
server_key_iv: "49dacd03d79e3082cbbcc6f338ab1340"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.643850 719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:51.644716 734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:51.644932 719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:51.645042 719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "c1855aa087f44db2a1ec2d27cefc02d4"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "bae3d4881b05bec936873b99dd745301"
server_key_iv: "49dacd03d79e3082cbbcc6f338ab1340"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.645135 719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:51.654012 719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:51.656395 719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:51.656606 719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:51.656898 719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:51.657264 719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:51.657338 719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:51.657423 719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:51.657455 719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:51.664991 719 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:34301
I20260812 00:12:51.665041 853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:34301 every 8 connection(s)
I20260812 00:12:51.665469 719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:12:51 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:51.675506 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 719
I20260812 00:12:51.675650 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:12:51.675877 16783 external_mini_cluster.cc:1428] Setting key 90c9fea2312f94e31cad11b3f75e792b
I20260812 00:12:51.676158 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:51.666326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:47929 (local address 127.16.99.254:37271)
0812 00:12:51.666416 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:51.666419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:51.666645 (+ 226us) server_negotiation.cc:407] Connection header received
0812 00:12:51.667156 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:51.667158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:51.667173 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:51.667223 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:51.668200 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.668810 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.669936 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.670149 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.670210 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:51.672869 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:51.672871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:51.672871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:51.672881 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:51.674569 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:51.674976 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:51.674977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:51.674977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:51.675054 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:51.675377 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:51.675379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:51.675379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:51.675536 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:51.675619 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:51.675968 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 00:12:51.676085 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 00:12:51.676469 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:51.666301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:47929)
0812 00:12:51.666509 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:51.666547 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:51.667055 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:51.667293 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:51.667297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:51.667891 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:51.668094 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:51.668098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.668941 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.668942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:51.669821 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:51.669824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.670457 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.670458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:51.670591 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:51.671244 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:51.671253 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:51.672761 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:51.674646 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:51.674653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:51.674655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:51.674905 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:51.675138 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:51.675139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:51.675139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:51.675269 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:51.675636 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:51.675644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:51.675817 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 00:12:51.675912 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 00:12:51.676155 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":123,"threads_started":1}
I20260812 00:12:51.677050 854 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37271
I20260812 00:12:51.677151 854 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:51.677356 854 heartbeater.cc:511] Master 127.16.99.254:37271 requested a full tablet report, sending...
I20260812 00:12:51.677944 652 ts_manager.cc:194] Registered new tserver with Master: c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301)
I20260812 00:12:51.678822 652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:47929
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:12:51.706398 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37271
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:51.796080 860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:51.796310 860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:51.796337 860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:51.797510 860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:51.797559 860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:51.797586 860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:12:51.799088 860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:51.799319 860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:51.799510 860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:51.805419 869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:51.805400 866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:51.805400 865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:51.805512 860 server_base.cc:1034] running on GCE node
I20260812 00:12:51.805912 860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:51.806232 860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:51.807518 860 hybrid_clock.cc:648] HybridClock initialized: now 1786493571807517 us; error 66 us; skew 500 ppm
Aug 12 00:12:51 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:51.810581 860 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:12:51.811151 860 webserver.cc:460] Webserver started at http://127.16.99.194:39435/ using document root <none> and password file <none>
I20260812 00:12:51.811429 860 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:51.811479 860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:51.811628 860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:51.812752 860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "40d5733edead4de0897ed0741b8e12fc"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "e652d4a1965001f418c02ef5548b48fe"
server_key_iv: "c0a11467db348559f1f0c056e3deb130"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.813126 860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "40d5733edead4de0897ed0741b8e12fc"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "e652d4a1965001f418c02ef5548b48fe"
server_key_iv: "c0a11467db348559f1f0c056e3deb130"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.814695 860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:12:51.815526 876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:51.815959 860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:12:51.816287 860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "40d5733edead4de0897ed0741b8e12fc"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "e652d4a1965001f418c02ef5548b48fe"
server_key_iv: "c0a11467db348559f1f0c056e3deb130"
server_key_version: "encryptionkey@0"
I20260812 00:12:51.816388 860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:51.848781 860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:51.851500 860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:51.851691 860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:51.851959 860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:51.852294 860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:51.852351 860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:51.852394 860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:51.852419 860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:51.859109 860 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:35663
I20260812 00:12:51.859158 999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:35663 every 8 connection(s)
I20260812 00:12:51.859520 860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:12:51.863600 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 860
I20260812 00:12:51.863739 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:12:51.863955 16783 external_mini_cluster.cc:1428] Setting key cc78fe8bbc7a2bde32ea04df7ea162d4
Aug 12 00:12:51 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493571, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
I20260812 00:12:51.877328 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:51.860751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:55647 (local address 127.16.99.254:37271)
0812 00:12:51.860850 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:51.860853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:51.861083 (+ 230us) server_negotiation.cc:407] Connection header received
0812 00:12:51.861913 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:51.861915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:51.861930 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:51.861982 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:51.863576 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.864413 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.868209 (+ 3796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.868502 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.868594 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:51.871836 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:51.871838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:51.871840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:51.871856 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:51.875121 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:51.875888 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:51.875891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:51.875892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:51.875976 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:51.876426 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:51.876428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:51.876428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:51.876648 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:51.876739 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:51.877106 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 00:12:51.877218 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 00:12:51.877655 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:51.860377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:55647)
0812 00:12:51.860881 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:51.860942 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:51.861747 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:51.862081 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:51.862086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:51.863093 (+ 1007us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:51.863452 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:51.863461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.864522 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.864523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:51.866588 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:51.866592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:51.868618 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:51.868619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:51.868769 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:51.869438 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:51.869449 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:51.871656 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:51.875256 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:51.875273 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:51.875277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:51.875745 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:51.876103 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:51.876105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:51.876106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:51.876310 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:51.876734 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:51.876747 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:51.876936 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 00:12:51.877064 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 00:12:51.877279 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":94,"threads_started":1}
I20260812 00:12:51.878393 1000 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37271
I20260812 00:12:51.878561 1000 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
I20260812 00:12:51.878799 1000 heartbeater.cc:511] Master 127.16.99.254:37271 requested a full tablet report, sending...
I20260812 00:12:51.879359 652 ts_manager.cc:194] Registered new tserver with Master: 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663)
I20260812 00:12:51.879813 652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:55647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:12:51.902158 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37271
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:12:52.002892 1005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:12:52.003144 1005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:12:52.003171 1005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:12:52.004350 1005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:12:52.004387 1005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:12:52.004413 1005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:12:52.006063 1005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:12:52.006299 1005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:12:52.006563 1005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:12:52.012389 1011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:52.012322 1013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:12:52.012437 1010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:12:52.012854 1005 server_base.cc:1034] running on GCE node
I20260812 00:12:52.013108 1005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:12:52.013427 1005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:12:52.014618 1005 hybrid_clock.cc:648] HybridClock initialized: now 1786493572014582 us; error 73 us; skew 500 ppm
Aug 12 00:12:52 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493572, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:12:52.017947 1005 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:12:52.018620 1005 webserver.cc:460] Webserver started at http://127.16.99.195:45157/ using document root <none> and password file <none>
I20260812 00:12:52.018893 1005 fs_manager.cc:359] Metadata directory not provided
I20260812 00:12:52.018944 1005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:12:52.019095 1005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:12:52.020366 1005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "b99a73f9649a4ed2a30c950accf137dd"
format_stamp: "Formatted at 2026-08-12 00:12:52 on dist-test-slave-zpfg"
server_key: "ceb536b6fb4d6e86197583ac303b50ad"
server_key_iv: "7dc8dabde636249584670bfb827213fb"
server_key_version: "encryptionkey@0"
I20260812 00:12:52.020829 1005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "b99a73f9649a4ed2a30c950accf137dd"
format_stamp: "Formatted at 2026-08-12 00:12:52 on dist-test-slave-zpfg"
server_key: "ceb536b6fb4d6e86197583ac303b50ad"
server_key_iv: "7dc8dabde636249584670bfb827213fb"
server_key_version: "encryptionkey@0"
I20260812 00:12:52.022535 1005 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260812 00:12:52.023699 1022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:52.024072 1005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:12:52.024489 1005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "b99a73f9649a4ed2a30c950accf137dd"
format_stamp: "Formatted at 2026-08-12 00:12:52 on dist-test-slave-zpfg"
server_key: "ceb536b6fb4d6e86197583ac303b50ad"
server_key_iv: "7dc8dabde636249584670bfb827213fb"
server_key_version: "encryptionkey@0"
I20260812 00:12:52.024600 1005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:12:52.044600 1005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:12:52.047997 1005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:12:52.048211 1005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:12:52.048573 1005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:12:52.049168 1005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:12:52.049242 1005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:52.049305 1005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:12:52.049336 1005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:12:52.057475 1005 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:39351
I20260812 00:12:52.057538 1143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:39351 every 8 connection(s)
I20260812 00:12:52.058007 1005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:12:52.059648 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 1005
I20260812 00:12:52.059757 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:12:52.059998 16783 external_mini_cluster.cc:1428] Setting key e49f1c9cd16744ac335fa9861a117a87
Aug 12 00:12:52 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493572, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:12:52.070030 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.059061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58523 (local address 127.16.99.254:37271)
0812 00:12:52.059338 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.059341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.059452 (+ 111us) server_negotiation.cc:407] Connection header received
0812 00:12:52.060135 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.060137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.060152 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.060230 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:52.061539 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.062070 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.063714 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.063929 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.064020 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.066752 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:52.066759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:52.066760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:52.066768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:52.068473 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:52.068935 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:52.068937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:52.068938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:52.069006 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:52.069301 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:52.069302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:52.069303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:52.069447 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:52.069528 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.069835 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.069932 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 00:12:52.070273 1146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.058952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:58523)
0812 00:12:52.059246 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.059309 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:52.059976 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:52.060327 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:52.060331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:52.061194 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:12:52.061425 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.061431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.062183 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.062184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.063598 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.063602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.064037 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.064038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.064195 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.064827 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:12:52.064836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:12:52.066625 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:12:52.068573 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:52.068586 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:52.068589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:52.068850 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:52.069087 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:12:52.069087 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:12:52.069088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:12:52.069229 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:12:52.069518 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:12:52.069526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:12:52.069693 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 00:12:52.069810 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:12:52.069991 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":125,"threads_started":1}
I20260812 00:12:52.070969 1144 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37271
I20260812 00:12:52.071096 1144 heartbeater.cc:464] Registering TS with master...
I20260812 00:12:52.071334 1144 heartbeater.cc:511] Master 127.16.99.254:37271 requested a full tablet report, sending...
I20260812 00:12:52.071806 652 ts_manager.cc:194] Registered new tserver with Master: b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
I20260812 00:12:52.072255 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:12:52.072306 652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:58523
I20260812 00:12:52.081110 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.074277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.16.99.254:37271)
0812 00:12:52.074413 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.074416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.074518 (+ 102us) server_negotiation.cc:407] Connection header received
0812 00:12:52.074629 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.074629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.074639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.074693 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:12:52.075367 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.075851 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.076794 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.076984 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.077040 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.078439 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:12:52.078441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:12:52.078441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:12:52.078447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:12:52.079923 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:52.080366 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:52.080367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:52.080367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:52.080422 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:12:52.080606 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:12:52.080606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:12:52.080606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:12:52.080762 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:12:52.080841 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.080973 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.081040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 00:12:52.083235 652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:12:52.084884 652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:12:52.092839 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.087898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.16.99.193:34301)
0812 00:12:52.088070 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.088077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.088424 (+ 347us) server_negotiation.cc:407] Connection header received
0812 00:12:52.088472 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.088474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.088492 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.088574 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:52.088885 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.090070 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.091566 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.092330 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.092410 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.092472 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.092693 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.092739 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260812 00:12:52.092844 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.088221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:35663 (local address 127.0.0.1:49070)
0812 00:12:52.088537 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.088548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:52.088594 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:52.089097 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:52.089098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:52.089100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:52.089139 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.089141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.090414 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.090415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.091333 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.091335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.092449 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.092450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.092556 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.092576 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 00:12:52.092669 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:12:52.092753 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":34,"thread_start_us":103,"threads_started":1}
I20260812 00:12:52.092936 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.087780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34301 (local address 127.0.0.1:50822)
0812 00:12:52.088169 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.088205 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:52.088320 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:52.088681 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:52.088684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:52.088687 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:52.088811 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.088814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.090194 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.090195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.091456 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.091459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.092442 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.092443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.092556 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.092576 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 00:12:52.092669 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:12:52.092819 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":27,"thread_start_us":77,"threads_started":1}
I20260812 00:12:52.093357 1161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.088489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.16.99.194:35663)
0812 00:12:52.088910 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.088914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.088924 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:52.088972 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.088974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.088990 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.089075 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:52.089222 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.090292 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.091566 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.092330 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.092410 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.092472 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.093029 (+ 557us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.093262 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":91,"threads_started":1}
I20260812 00:12:52.094012 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.087996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:39351 (local address 127.0.0.1:59612)
0812 00:12:52.088310 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.088321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:52.088378 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:52.088835 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:52.088836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:52.088838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:52.088877 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.088879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.090003 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.090004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.091457 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.091459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.093696 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.093698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.093790 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.093794 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:12:52.093882 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 00:12:52.093934 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":50,"thread_start_us":110,"threads_started":1}
I20260812 00:12:52.093863 784 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1fbb717bd14ff48ba619b2001fa3e7 (DEFAULT_TABLE table=test-table [id=8615b12fc3e343bb9c6a2ddf78403ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:52.094056 929 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1fbb717bd14ff48ba619b2001fa3e7 (DEFAULT_TABLE table=test-table [id=8615b12fc3e343bb9c6a2ddf78403ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:52.094031 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.088122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.16.99.195:39351)
0812 00:12:52.088344 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.088349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.088369 (+ 20us) server_negotiation.cc:407] Connection header received
0812 00:12:52.088471 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.088473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.088713 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.088961 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:52.089024 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.089879 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.092516 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.093592 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.093700 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.093772 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.093900 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.093945 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260812 00:12:52.094319 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1fbb717bd14ff48ba619b2001fa3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:52.094480 929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1fbb717bd14ff48ba619b2001fa3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:52.094971 1073 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1fbb717bd14ff48ba619b2001fa3e7 (DEFAULT_TABLE table=test-table [id=8615b12fc3e343bb9c6a2ddf78403ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:12:52.095311 1073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1fbb717bd14ff48ba619b2001fa3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:12:52.097397 1162 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap starting.
I20260812 00:12:52.098315 1162 tablet_bootstrap.cc:654] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:52.098562 1163 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Bootstrap starting.
I20260812 00:12:52.098845 1164 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Bootstrap starting.
I20260812 00:12:52.099067 1162 log.cc:824] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:52.099267 1163 tablet_bootstrap.cc:654] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:52.099597 1164 tablet_bootstrap.cc:654] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Neither blocks nor log segments found. Creating new log.
I20260812 00:12:52.099840 1163 log.cc:824] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:52.100030 1162 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: No bootstrap required, opened a new log
I20260812 00:12:52.100095 1162 ts_tablet_manager.cc:1397] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:12:52.100162 1164 log.cc:824] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Log is configured to *not* fsync() on all Append() calls
I20260812 00:12:52.100757 1163 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: No bootstrap required, opened a new log
I20260812 00:12:52.100818 1163 ts_tablet_manager.cc:1397] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:52.101154 1164 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: No bootstrap required, opened a new log
I20260812 00:12:52.101207 1164 ts_tablet_manager.cc:1397] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:12:52.102289 1162 raft_consensus.cc:348] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.102528 1162 raft_consensus.cc:374] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:52.102569 1162 raft_consensus.cc:729] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1855aa087f44db2a1ec2d27cefc02d4, State: Initialized, Role: FOLLOWER
I20260812 00:12:52.102703 1162 consensus_queue.cc:260] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.102870 1164 raft_consensus.cc:348] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.103036 1162 ts_tablet_manager.cc:1428] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:52.103076 854 heartbeater.cc:503] Master 127.16.99.254:37271 was elected leader, sending a full tablet report...
I20260812 00:12:52.103085 1164 raft_consensus.cc:374] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:52.103127 1164 raft_consensus.cc:729] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99a73f9649a4ed2a30c950accf137dd, State: Initialized, Role: FOLLOWER
I20260812 00:12:52.103049 1163 raft_consensus.cc:348] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.103230 1163 raft_consensus.cc:374] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:12:52.103263 1163 raft_consensus.cc:729] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d5733edead4de0897ed0741b8e12fc, State: Initialized, Role: FOLLOWER
I20260812 00:12:52.103293 1164 consensus_queue.cc:260] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.103376 1163 consensus_queue.cc:260] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.103621 1164 ts_tablet_manager.cc:1428] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:12:52.103657 1163 ts_tablet_manager.cc:1428] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:12:52.103685 1000 heartbeater.cc:503] Master 127.16.99.254:37271 was elected leader, sending a full tablet report...
I20260812 00:12:52.103749 1144 heartbeater.cc:503] Master 127.16.99.254:37271 was elected leader, sending a full tablet report...
W20260812 00:12:52.110201 1001 tablet.cc:2369] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:52.155083 1170 raft_consensus.cc:482] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:12:52.155376 1170 raft_consensus.cc:504] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.155807 1170 leader_election.cc:288] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
I20260812 00:12:52.159830 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.156010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34301 (local address 127.16.99.194:53827)
0812 00:12:52.156370 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.156381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:52.156447 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:52.156663 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:52.156664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:52.156666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:52.156712 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.156713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.157500 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.157500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.158717 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.158721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.159535 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.159536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.159594 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.159599 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:12:52.159679 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:12:52.159723 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":108,"thread_start_us":135,"threads_started":1}
I20260812 00:12:52.159847 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.156146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:53827 (local address 127.16.99.193:34301)
0812 00:12:52.156196 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.156198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.156503 (+ 305us) server_negotiation.cc:407] Connection header received
0812 00:12:52.156534 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.156535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.156548 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.156653 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:52.156772 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.157433 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.158848 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.159437 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.159529 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.159540 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.159705 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.159752 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:12:52.159931 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.156008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:39351 (local address 127.16.99.194:49139)
0812 00:12:52.156057 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.156083 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 00:12:52.156130 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:12:52.156343 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:12:52.156344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:12:52.156347 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:12:52.156428 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.156430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.157377 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.157378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.158717 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:12:52.158721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.159667 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.159668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:12:52.159739 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.159742 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:12:52.159819 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 00:12:52.159864 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260812 00:12:52.159935 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.156127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:49139 (local address 127.16.99.195:39351)
0812 00:12:52.156186 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.156188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.156200 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:12:52.156234 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.156235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.156245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.156325 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:12:52.156520 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.157297 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.158848 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.159582 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.159658 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.159668 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.159823 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.159866 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 00:12:52.160317 1093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbb717bd14ff48ba619b2001fa3e7" candidate_uuid: "40d5733edead4de0897ed0741b8e12fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b99a73f9649a4ed2a30c950accf137dd" is_pre_election: true
I20260812 00:12:52.160374 805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbb717bd14ff48ba619b2001fa3e7" candidate_uuid: "40d5733edead4de0897ed0741b8e12fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" is_pre_election: true
I20260812 00:12:52.160511 1093 raft_consensus.cc:2454] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d5733edead4de0897ed0741b8e12fc in term 0.
I20260812 00:12:52.160555 805 raft_consensus.cc:2454] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d5733edead4de0897ed0741b8e12fc in term 0.
I20260812 00:12:52.160827 877 leader_election.cc:302] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d5733edead4de0897ed0741b8e12fc, c1855aa087f44db2a1ec2d27cefc02d4; no voters:
I20260812 00:12:52.160989 1170 raft_consensus.cc:2781] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:12:52.161049 1170 raft_consensus.cc:482] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:12:52.161132 1170 raft_consensus.cc:3037] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:52.161911 1170 raft_consensus.cc:504] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.162148 1170 leader_election.cc:288] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [CANDIDATE]: Term 1 election: Requested vote from peers c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
I20260812 00:12:52.162420 1093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbb717bd14ff48ba619b2001fa3e7" candidate_uuid: "40d5733edead4de0897ed0741b8e12fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b99a73f9649a4ed2a30c950accf137dd"
I20260812 00:12:52.162436 805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbb717bd14ff48ba619b2001fa3e7" candidate_uuid: "40d5733edead4de0897ed0741b8e12fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1855aa087f44db2a1ec2d27cefc02d4"
I20260812 00:12:52.162539 805 raft_consensus.cc:3037] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:52.162523 1093 raft_consensus.cc:3037] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:12:52.163335 805 raft_consensus.cc:2454] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d5733edead4de0897ed0741b8e12fc in term 1.
I20260812 00:12:52.163386 1093 raft_consensus.cc:2454] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d5733edead4de0897ed0741b8e12fc in term 1.
I20260812 00:12:52.163555 877 leader_election.cc:302] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d5733edead4de0897ed0741b8e12fc, c1855aa087f44db2a1ec2d27cefc02d4; no voters:
I20260812 00:12:52.163671 1170 raft_consensus.cc:2781] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:12:52.163856 1170 raft_consensus.cc:686] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 1 LEADER]: Becoming Leader. State: Replica: 40d5733edead4de0897ed0741b8e12fc, State: Running, Role: LEADER
I20260812 00:12:52.163985 1170 consensus_queue.cc:237] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:12:52.165119 651 catalog_manager.cc:5495] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 40d5733edead4de0897ed0741b8e12fc (127.16.99.194). New cstate: current_term: 1 leader_uuid: "40d5733edead4de0897ed0741b8e12fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } health_report { overall_health: HEALTHY } } }
W20260812 00:12:52.166371 855 tablet.cc:2369] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:12:52.172816 1161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.170012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.16.99.194:35663)
0812 00:12:52.170057 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:12:52.170059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:12:52.170069 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:12:52.170111 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:12:52.170112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:12:52.170140 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:12:52.170212 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:12:52.170388 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.170879 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.171749 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:12:52.171980 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:12:52.172038 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:12:52.172356 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:12:52.172483 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:12:52.172590 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 00:12:52.172670 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 00:12:52.172695 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:12:52.175146 1093 raft_consensus.cc:1261] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 1 FOLLOWER]: Refusing update from remote peer 40d5733edead4de0897ed0741b8e12fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:52.175184 805 raft_consensus.cc:1261] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Refusing update from remote peer 40d5733edead4de0897ed0741b8e12fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:12:52.175506 1170 consensus_queue.cc:1035] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:52.175740 1172 consensus_queue.cc:1035] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:12:52.179319 1177 mvcc.cc:205] Tried to move back new op lower bound from 7317477671626412032 to 7317477671584604160. Current Snapshot: MvccSnapshot[applied={T|T < 7317477671626412032}]
I20260812 00:12:52.181861 651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:52.181985 651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:12:52.182960 651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Sending DeleteTablet for 3 replicas of tablet 2b1fbb717bd14ff48ba619b2001fa3e7
I20260812 00:12:52.183406 1073 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1fbb717bd14ff48ba619b2001fa3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:59612
I20260812 00:12:52.183423 929 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1fbb717bd14ff48ba619b2001fa3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:49070
I20260812 00:12:52.183568 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1fbb717bd14ff48ba619b2001fa3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:50822
I20260812 00:12:52.183851 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 719
I20260812 00:12:52.183873 1188 tablet_replica.cc:330] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: stopping tablet replica
I20260812 00:12:52.190537 1187 tablet_replica.cc:330] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: stopping tablet replica
I20260812 00:12:52.190708 1187 raft_consensus.cc:2229] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:12:52.190886 1187 raft_consensus.cc:2258] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:52.190980 1186 tablet_replica.cc:330] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: stopping tablet replica
I20260812 00:12:52.191177 1186 raft_consensus.cc:2229] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:12:52.191368 1186 raft_consensus.cc:2258] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:12:52.191450 1187 ts_tablet_manager.cc:1905] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:12:52.191998 1186 ts_tablet_manager.cc:1905] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 00:12:52.192443 636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:34301 (error 108)
I20260812 00:12:52.193398 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.193240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34301 (local address 127.0.0.1:50830)
0812 00:12:52.193312 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.193355 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:34301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 00:12:52.193427 1187 ts_tablet_manager.cc:1918] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:52.193504 1187 log.cc:1196] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/2b1fbb717bd14ff48ba619b2001fa3e7
W20260812 00:12:52.193536 636 catalog_manager.cc:4584] TS c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b1fbb717bd14ff48ba619b2001fa3e7: Network error: Client connection negotiation failed: client connection to 127.16.99.193:34301: connect: Connection refused (error 111)
I20260812 00:12:52.193748 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 860
I20260812 00:12:52.197028 1186 ts_tablet_manager.cc:1918] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:12:52.197150 1186 log.cc:1196] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/2b1fbb717bd14ff48ba619b2001fa3e7
I20260812 00:12:52.197475 1186 ts_tablet_manager.cc:1939] T 2b1fbb717bd14ff48ba619b2001fa3e7 P b99a73f9649a4ed2a30c950accf137dd: Deleting consensus metadata
I20260812 00:12:52.198107 637 catalog_manager.cc:4841] TS b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): tablet 2b1fbb717bd14ff48ba619b2001fa3e7 (table test-table [id=8615b12fc3e343bb9c6a2ddf78403ac2]) successfully deleted
I20260812 00:12:52.201395 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:12:52.201239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:35663 (local address 127.0.0.1:49078)
0812 00:12:52.201333 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 00:12:52.201360 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.194:35663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 00:12:52.201581 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 1005
I20260812 00:12:52.208144 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 620
I20260812 00:13:12.215751 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37271
--webserver_interface=127.16.99.254
--webserver_port=33353
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:12.303128 1190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:12.303403 1190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:12.303458 1190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:12.304730 1190 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 00:13:12.304809 1190 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 00:13:12.304831 1190 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 00:13:12.304852 1190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:12.304879 1190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:12.304913 1190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:12.304939 1190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:12.304967 1190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:12.306607 1190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=33353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:12.306924 1190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:12.307299 1190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:12.318229 1203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:12.318187 1199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:12.318173 1201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:12.318564 1190 server_base.cc:1034] running on GCE node
I20260812 00:13:12.318861 1190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:12.319170 1190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:12.320333 1190 hybrid_clock.cc:648] HybridClock initialized: now 1786493592320309 us; error 41 us; skew 500 ppm
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:12.323334 1190 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:13:12.323899 1190 webserver.cc:460] Webserver started at http://127.16.99.254:33353/ using document root <none> and password file <none>
I20260812 00:13:12.324186 1190 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:12.324293 1190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:12.326009 1190 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:12.327028 1210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:12.327278 1190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:13:12.327418 1190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "57e08f90af1c4069a60f4c36a30ac556"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "f34cb93ff3e90c738a6aaef11dc7d4d3"
server_key_iv: "ae34b117b83f42189c7a03321cece8e8"
server_key_version: "encryptionkey@0"
I20260812 00:13:12.327925 1190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:12.341251 1190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:12.343624 1190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:12.343830 1190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:12.348542 1190 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:37271
I20260812 00:13:12.348587 1265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:37271 every 8 connection(s)
I20260812 00:13:12.349064 1190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:12.350906 1266 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:13:12.351692 1266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Bootstrap starting.
I20260812 00:13:12.352670 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 1190
I20260812 00:13:12.352996 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:34301
--local_ip_for_outbound_sockets=127.16.99.193
--tserver_master_addrs=127.16.99.254:37271
--webserver_port=44291
--webserver_interface=127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:13:12.362020 1266 log.cc:824] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:12.364341 1266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:13:12.364746 1266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Bootstrap complete.
I20260812 00:13:12.367552 1266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:12.367869 1266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e08f90af1c4069a60f4c36a30ac556, State: Initialized, Role: FOLLOWER
I20260812 00:13:12.368089 1266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:12.368183 1266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:12.368254 1266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:12.368315 1266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 1 FOLLOWER]: Advancing to term 2
I20260812 00:13:12.369544 1266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:12.369701 1266 leader_election.cc:302] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57e08f90af1c4069a60f4c36a30ac556; no voters:
I20260812 00:13:12.369954 1266 leader_election.cc:288] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 00:13:12.370002 1270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 00:13:12.370148 1270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 LEADER]: Becoming Leader. State: Replica: 57e08f90af1c4069a60f4c36a30ac556, State: Running, Role: LEADER
I20260812 00:13:12.370309 1266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:12.370328 1270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:12.370786 1272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57e08f90af1c4069a60f4c36a30ac556. Latest consensus state: current_term: 2 leader_uuid: "57e08f90af1c4069a60f4c36a30ac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } } }
I20260812 00:13:12.370959 1272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:12.371160 1271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57e08f90af1c4069a60f4c36a30ac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } } }
I20260812 00:13:12.371284 1271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:12.371429 1278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:12.372278 1278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:12.372385 1278 catalog_manager.cc:1259] Loaded cluster ID: 3b39026ff976421eab4e3252ab4545ce
I20260812 00:13:12.372465 1278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:12.373839 1278 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:12.374035 1278 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Loaded TSK: 0
I20260812 00:13:12.384267 1278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Generated new TSK 1
I20260812 00:13:12.384400 1278 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:13:12.444307 1268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:12.444540 1268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:12.444571 1268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:12.445832 1268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:12.445883 1268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:12.445912 1268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:12.447523 1268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:34301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=44291
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:12.447772 1268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:12.448035 1268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:12.453444 1294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:12.453462 1295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:12.453464 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:12.453703 1268 server_base.cc:1034] running on GCE node
I20260812 00:13:12.453980 1268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:12.454560 1268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:12.455775 1268 hybrid_clock.cc:648] HybridClock initialized: now 1786493592455741 us; error 52 us; skew 500 ppm
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:12.458673 1268 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:13:12.459205 1268 webserver.cc:460] Webserver started at http://127.16.99.193:44291/ using document root <none> and password file <none>
I20260812 00:13:12.459460 1268 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:12.459544 1268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:12.461001 1268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:12.461735 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:12.461913 1268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:12.462023 1268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "c1855aa087f44db2a1ec2d27cefc02d4"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "bae3d4881b05bec936873b99dd745301"
server_key_iv: "49dacd03d79e3082cbbcc6f338ab1340"
server_key_version: "encryptionkey@0"
I20260812 00:13:12.462450 1268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:12.470605 1268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:12.473309 1268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:12.473477 1268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:12.473773 1268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:12.474555 1313 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:13:12.476357 1268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:13:12.476413 1268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 00:13:12.476450 1268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:13:12.477152 1268 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:13:12.477200 1268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 00:13:12.477257 1313 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap starting.
I20260812 00:13:12.485812 1268 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:34301
I20260812 00:13:12.485913 1431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:34301 every 8 connection(s)
I20260812 00:13:12.486455 1268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:12.489964 1313 log.cc:824] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:12.492697 1313 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:13:12.493105 1313 tablet_bootstrap.cc:492] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap complete.
I20260812 00:13:12.494555 1313 ts_tablet_manager.cc:1397] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.014s
I20260812 00:13:12.495700 1313 raft_consensus.cc:348] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:13:12.495976 1313 raft_consensus.cc:729] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1855aa087f44db2a1ec2d27cefc02d4, State: Initialized, Role: FOLLOWER
I20260812 00:13:12.496192 1313 consensus_queue.cc:260] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } }
I20260812 00:13:12.496565 1313 ts_tablet_manager.cc:1428] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:13:12.496688 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 1268
I20260812 00:13:12.497004 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:35663
--local_ip_for_outbound_sockets=127.16.99.194
--tserver_master_addrs=127.16.99.254:37271
--webserver_port=39435
--webserver_interface=127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:12.504587 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.487525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:33951)
0812 00:13:12.490279 (+ 2754us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.490372 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.491104 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.491719 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.491724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.492157 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:12.492489 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.492498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.494839 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.494841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.496293 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.496297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.497263 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.497264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.497415 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.498167 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:12.498177 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:12.499841 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:12.502268 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.502277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.502279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.502678 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.503092 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.503093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.503094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.503272 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.503841 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:12.503850 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:12.504026 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.504154 (+ 128us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.504351 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2690,"thread_start_us":114,"threads_started":1}
I20260812 00:13:12.504721 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.487636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:33951 (local address 127.16.99.254:37271)
0812 00:13:12.490426 (+ 2790us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.490437 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.490492 (+ 55us) server_negotiation.cc:407] Connection header received
0812 00:13:12.491400 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.491408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.491474 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.491641 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:12.492660 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.494694 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.496426 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.497158 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.497291 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.499963 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:12.499968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:12.499968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:12.499980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:12.502155 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.502840 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.502844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.502844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.502991 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.503373 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.503374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.503374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.503729 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:12.503840 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.504270 (+ 430us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.504438 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2723,"thread_start_us":98,"threads_started":1}
I20260812 00:13:12.506319 1432 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37271
I20260812 00:13:12.506491 1432 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:12.506742 1432 heartbeater.cc:511] Master 127.16.99.254:37271 requested a full tablet report, sending...
I20260812 00:13:12.507642 1228 ts_manager.cc:194] Registered new tserver with Master: c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301)
I20260812 00:13:12.509049 1228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:33951
I20260812 00:13:12.510169 1432 heartbeater.cc:503] Master 127.16.99.254:37271 was elected leader, sending a full tablet report...
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.193@KRBTEST.COM
I20260812 00:13:12.520960 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.508404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34301 (local address 127.0.0.1:51964)
0812 00:13:12.508931 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.508974 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.509052 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.510562 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.510569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.511171 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:12.511290 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.511293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.512808 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.512809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.513894 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.513898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.514686 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.514687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.514774 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.515382 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:12.515388 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:12.517192 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:12.519572 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.519579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.519581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.519862 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.520122 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.520123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.520123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.520193 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.520538 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:12.520545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:12.520602 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.520694 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.520835 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":116,"threads_started":1}
I20260812 00:13:12.521005 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.508721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.16.99.193:34301)
0812 00:13:12.509202 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.509207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.509217 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:13:12.509254 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.509256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.509271 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.510492 (+ 1221us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:12.511398 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.512694 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.514005 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.514516 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.514584 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.517294 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:12.517304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:12.517305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:12.517314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:12.519471 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.519959 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.519963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.519963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.520035 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.520259 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.520260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.520261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.520448 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:12.520530 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.520752 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.520860 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":100,"threads_started":1}
I20260812 00:13:12.521777 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1fbb717bd14ff48ba619b2001fa3e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:12:52 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51964
I20260812 00:13:12.522488 1441 tablet_replica.cc:330] stopping tablet replica
I20260812 00:13:12.522604 1441 raft_consensus.cc:2229] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:12.522699 1441 raft_consensus.cc:2258] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:12.523062 1441 ts_tablet_manager.cc:1905] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:12.525549 1441 ts_tablet_manager.cc:1918] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:12.525633 1441 log.cc:1196] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal/wals/2b1fbb717bd14ff48ba619b2001fa3e7
I20260812 00:13:12.525876 1441 ts_tablet_manager.cc:1939] T 2b1fbb717bd14ff48ba619b2001fa3e7 P c1855aa087f44db2a1ec2d27cefc02d4: Deleting consensus metadata
I20260812 00:13:12.526563 1212 catalog_manager.cc:4841] TS c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301): tablet 2b1fbb717bd14ff48ba619b2001fa3e7 (table test-table [id=8615b12fc3e343bb9c6a2ddf78403ac2]) successfully deleted
W20260812 00:13:12.589570 1438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:12.589793 1438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:12.589818 1438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:12.590974 1438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:12.591017 1438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:12.591043 1438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:12.592478 1438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:35663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=39435
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:12.592679 1438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:12.592865 1438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:12.598579 1449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:12.598713 1438 server_base.cc:1034] running on GCE node
W20260812 00:13:12.598598 1447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:12.598584 1446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:12.599079 1438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:12.599375 1438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:12.600564 1438 hybrid_clock.cc:648] HybridClock initialized: now 1786493592600555 us; error 37 us; skew 500 ppm
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:12.603472 1438 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:13:12.603951 1438 webserver.cc:460] Webserver started at http://127.16.99.194:39435/ using document root <none> and password file <none>
I20260812 00:13:12.604228 1438 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:12.604342 1438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:12.605867 1438 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:12.606736 1456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:12.606961 1438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:12.607081 1438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "40d5733edead4de0897ed0741b8e12fc"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "e652d4a1965001f418c02ef5548b48fe"
server_key_iv: "c0a11467db348559f1f0c056e3deb130"
server_key_version: "encryptionkey@0"
I20260812 00:13:12.607558 1438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:12.616400 1438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:12.618613 1438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:12.618800 1438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:12.619099 1438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:12.619580 1463 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260812 00:13:12.621155 1463 ts_tablet_manager.cc:1852] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260812 00:13:12.621469 1463 ts_tablet_manager.cc:1905] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:12.623813 1463 ts_tablet_manager.cc:1918] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:12.623980 1463 ts_tablet_manager.cc:1939] T 2b1fbb717bd14ff48ba619b2001fa3e7 P 40d5733edead4de0897ed0741b8e12fc: Deleting consensus metadata
I20260812 00:13:12.624236 1438 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260812 00:13:12.624331 1438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260812 00:13:12.624384 1438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:12.624415 1438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:12.631821 1438 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:35663
I20260812 00:13:12.631911 1579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:35663 every 8 connection(s)
I20260812 00:13:12.632303 1438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:13:12.634933 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 1438
I20260812 00:13:12.635300 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:39351
--local_ip_for_outbound_sockets=127.16.99.195
--tserver_master_addrs=127.16.99.254:37271
--webserver_port=45157
--webserver_interface=127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:12.643721 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.633145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:51641 (local address 127.16.99.254:37271)
0812 00:13:12.633192 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.633197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.633469 (+ 272us) server_negotiation.cc:407] Connection header received
0812 00:13:12.634035 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.634037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.634051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.634116 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:12.634967 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.635729 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.637338 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.637562 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.637625 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.640523 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:12.640524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:12.640525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:12.640541 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:12.642040 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.642654 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.642656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.642656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.642723 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.643007 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.643008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.643009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.643134 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:12.643209 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.643522 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.643624 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:13:12.643914 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.633092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:51641)
0812 00:13:12.633306 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.633355 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.633942 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.634198 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.634203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.634551 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:12.634840 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.634846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.635890 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.635891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.637225 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.637229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.637666 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.637668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.637861 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.638606 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:12.638619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:12.640400 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:12.642160 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.642171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.642174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.642570 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.642802 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.642803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.642803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.642931 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.643212 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:12.643220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:12.643382 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.643488 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.643665 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":118,"threads_started":1}
I20260812 00:13:12.644505 1580 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37271
I20260812 00:13:12.644606 1580 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:12.644824 1580 heartbeater.cc:511] Master 127.16.99.254:37271 requested a full tablet report, sending...
I20260812 00:13:12.645218 1228 ts_manager.cc:194] Registered new tserver with Master: 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663)
I20260812 00:13:12.645618 1228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:51641
W20260812 00:13:12.734000 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:12.734298 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:12.734347 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:12.735816 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:12.735929 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:12.735981 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:12.737911 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:39351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=45157
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:12.738284 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:12.738610 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:12.744918 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:12.744911 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:12.745039 1583 server_base.cc:1034] running on GCE node
W20260812 00:13:12.744968 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:12.745497 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:12.745750 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:12.746902 1583 hybrid_clock.cc:648] HybridClock initialized: now 1786493592746888 us; error 31 us; skew 500 ppm
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:12.749871 1583 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:13:12.750406 1583 webserver.cc:460] Webserver started at http://127.16.99.195:45157/ using document root <none> and password file <none>
I20260812 00:13:12.750656 1583 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:12.750756 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:12.752199 1583 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:12.753010 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:12.753293 1583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:12.753407 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "b99a73f9649a4ed2a30c950accf137dd"
format_stamp: "Formatted at 2026-08-12 00:12:52 on dist-test-slave-zpfg"
server_key: "ceb536b6fb4d6e86197583ac303b50ad"
server_key_iv: "7dc8dabde636249584670bfb827213fb"
server_key_version: "encryptionkey@0"
I20260812 00:13:12.753785 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:12.783792 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:12.786543 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:12.786754 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:12.787058 1583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:12.787406 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:12.787470 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:12.787515 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:12.787555 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:12.794091 1583 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:39351
I20260812 00:13:12.794215 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:39351 every 8 connection(s)
I20260812 00:13:12.794605 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:13:12 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493592, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:12.803540 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 1583
I20260812 00:13:12.805471 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.795434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:58235 (local address 127.16.99.254:37271)
0812 00:13:12.795531 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.795534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.795932 (+ 398us) server_negotiation.cc:407] Connection header received
0812 00:13:12.796534 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.796535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.796549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.796632 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:12.797785 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.798305 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.799530 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.799728 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.799797 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.802191 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:12.802193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:12.802193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:12.802201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:12.803707 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.804287 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.804289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.804289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.804356 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.804709 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.804710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.804711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.804840 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:12.804905 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.805286 (+ 381us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.805376 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:13:12.805650 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.795441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:58235)
0812 00:13:12.795731 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.795790 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.796421 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.796698 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.796701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.797404 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:12.797655 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.797660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.798470 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.798471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.799440 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.799443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.799844 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.799845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.799972 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.800512 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:12.800521 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:12.802058 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:12.803791 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.803800 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.803803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.804126 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.804437 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:12.804438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:12.804438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:12.804597 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:12.804923 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:12.804931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:12.805114 (+ 183us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.805233 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.805418 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":132,"threads_started":1}
I20260812 00:13:12.806214 1721 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37271
I20260812 00:13:12.806375 1721 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:12.806587 1721 heartbeater.cc:511] Master 127.16.99.254:37271 requested a full tablet report, sending...
I20260812 00:13:12.807077 1228 ts_manager.cc:194] Registered new tserver with Master: b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
I20260812 00:13:12.807564 1228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:58235
I20260812 00:13:12.811266 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.804017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.16.99.254:37271)
0812 00:13:12.804189 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.804191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.804252 (+ 61us) server_negotiation.cc:407] Connection header received
0812 00:13:12.804345 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.804345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.804359 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.804421 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:12.805192 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.805978 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.807081 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.807425 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.807496 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.808806 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:12.808807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:12.808808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:12.808814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:12.810102 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.810520 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.810521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.810522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.810582 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:12.810796 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:12.810796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:12.810797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:12.810931 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:12.811070 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.811122 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.811195 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260812 00:13:12.812495 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:12.815626 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.812915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.16.99.254:37271)
0812 00:13:12.812984 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.812986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.813126 (+ 140us) server_negotiation.cc:407] Connection header received
0812 00:13:12.813158 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.813159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.813194 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.813247 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:12.813421 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.813898 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.814726 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.814937 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.814994 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.815222 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:12.815356 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:12.815468 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.815488 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.815521 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:13:12.815996 1228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:12.817828 1228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:12.820425 1360 tablet_service.cc:1467] Processing CreateTablet for tablet 1370e2812b1d4402af1a860cebbe819e (DEFAULT_TABLE table=test-table [id=e909f4bd7e384d45b96aed13d5f4abd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:12.820658 1360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1370e2812b1d4402af1a860cebbe819e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:12.822201 1313 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap starting.
I20260812 00:13:12.822587 1313 tablet_bootstrap.cc:654] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:12.823182 1313 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: No bootstrap required, opened a new log
I20260812 00:13:12.823239 1313 ts_tablet_manager.cc:1397] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:12.823603 1313 raft_consensus.cc:348] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.823699 1313 raft_consensus.cc:374] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:12.823758 1313 raft_consensus.cc:729] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1855aa087f44db2a1ec2d27cefc02d4, State: Initialized, Role: FOLLOWER
I20260812 00:13:12.823813 1313 consensus_queue.cc:260] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.823917 1313 ts_tablet_manager.cc:1428] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:12.824074 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.819999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:35663 (local address 127.0.0.1:44592)
0812 00:13:12.820040 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.820053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.820094 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.820495 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.820496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.820498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:12.820554 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.820556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.821734 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.821735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.822809 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.822812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.823803 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.823804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.823884 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.823910 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.823938 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.823986 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260812 00:13:12.824429 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.820142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.16.99.194:35663)
0812 00:13:12.820331 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.820337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.820348 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:12.820394 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.820396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.820413 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.820501 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:12.820601 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.821645 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.822894 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.823699 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.824195 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.824276 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.824304 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.824347 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260812 00:13:12.825050 1512 tablet_service.cc:1467] Processing CreateTablet for tablet 1370e2812b1d4402af1a860cebbe819e (DEFAULT_TABLE table=test-table [id=e909f4bd7e384d45b96aed13d5f4abd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:12.825423 1512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1370e2812b1d4402af1a860cebbe819e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:12.826872 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.822294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.16.99.195:39351)
0812 00:13:12.822572 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.822576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.822585 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:13:12.822617 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.822619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.822657 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.822741 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:12.823083 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.824129 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.825343 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.826446 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.826531 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.826597 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.826716 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.826763 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":156,"threads_started":1}
I20260812 00:13:12.826886 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.820096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:39351 (local address 127.0.0.1:34628)
0812 00:13:12.820306 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.820317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.820366 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.822880 (+ 2514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.822881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.822884 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:12.822932 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.822934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.824238 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.824239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.825240 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.825243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.826538 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.826539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.826610 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.826615 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.826698 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.826790 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":131,"threads_started":1}
I20260812 00:13:12.827785 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 1370e2812b1d4402af1a860cebbe819e (DEFAULT_TABLE table=test-table [id=e909f4bd7e384d45b96aed13d5f4abd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:12.828186 1463 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Bootstrap starting.
I20260812 00:13:12.828212 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1370e2812b1d4402af1a860cebbe819e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:12.829051 1463 tablet_bootstrap.cc:654] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:12.829334 1463 log.cc:824] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:12.830152 1463 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: No bootstrap required, opened a new log
I20260812 00:13:12.830209 1463 ts_tablet_manager.cc:1397] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:13:12.832109 1463 raft_consensus.cc:348] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.832386 1463 raft_consensus.cc:374] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:12.832444 1463 raft_consensus.cc:729] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d5733edead4de0897ed0741b8e12fc, State: Initialized, Role: FOLLOWER
I20260812 00:13:12.832587 1463 consensus_queue.cc:260] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.832733 1734 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Bootstrap starting.
I20260812 00:13:12.832959 1463 ts_tablet_manager.cc:1428] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:13:12.833001 1580 heartbeater.cc:503] Master 127.16.99.254:37271 was elected leader, sending a full tablet report...
I20260812 00:13:12.833494 1734 tablet_bootstrap.cc:654] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:12.834164 1734 log.cc:824] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:12.834973 1734 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: No bootstrap required, opened a new log
I20260812 00:13:12.835033 1734 ts_tablet_manager.cc:1397] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:13:12.836416 1734 raft_consensus.cc:348] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.836563 1734 raft_consensus.cc:374] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:12.836700 1734 raft_consensus.cc:729] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99a73f9649a4ed2a30c950accf137dd, State: Initialized, Role: FOLLOWER
I20260812 00:13:12.836833 1734 consensus_queue.cc:260] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.837162 1734 ts_tablet_manager.cc:1428] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 00:13:12.837198 1721 heartbeater.cc:503] Master 127.16.99.254:37271 was elected leader, sending a full tablet report...
W20260812 00:13:12.883013 1581 tablet.cc:2369] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:12.928767 1737 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:12.928944 1737 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.929400 1737 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301)
I20260812 00:13:12.933346 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.929716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:39353 (local address 127.16.99.193:34301)
0812 00:13:12.929766 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.929768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.929778 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:13:12.929823 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.929824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.929855 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.929934 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:12.930142 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.930759 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.932213 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.932993 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.933063 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.933138 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.933221 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.933270 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:13:12.933346 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.930004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:35407 (local address 127.16.99.194:35663)
0812 00:13:12.930073 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.930076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.930084 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:13:12.930114 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.930116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.930129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.930184 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:12.930384 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.930958 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.932333 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.932929 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.933012 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.933023 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.933225 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.933277 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 00:13:12.933446 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.929580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34301 (local address 127.16.99.195:39353)
0812 00:13:12.929619 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.929631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.929680 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.929955 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.929956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.929957 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:12.930061 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.930063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.930855 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.930856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.932100 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.932104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.933066 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.933067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.933139 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.933142 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.933202 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.933265 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":25}
I20260812 00:13:12.933642 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.929610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:35663 (local address 127.16.99.195:35407)
0812 00:13:12.929821 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.929830 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:12.929867 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:12.930212 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:12.930213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:12.930215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:12.930288 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.930290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.931013 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.931013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.932198 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:12.932200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.933050 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.933052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:12.933137 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.933142 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:12.933202 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:13:12.933250 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":136,"threads_started":1}
I20260812 00:13:12.933764 1381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1370e2812b1d4402af1a860cebbe819e" candidate_uuid: "b99a73f9649a4ed2a30c950accf137dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" is_pre_election: true
I20260812 00:13:12.934010 1381 raft_consensus.cc:2454] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b99a73f9649a4ed2a30c950accf137dd in term 0.
I20260812 00:13:12.934090 1532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1370e2812b1d4402af1a860cebbe819e" candidate_uuid: "b99a73f9649a4ed2a30c950accf137dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d5733edead4de0897ed0741b8e12fc" is_pre_election: true
I20260812 00:13:12.934252 1532 raft_consensus.cc:2454] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b99a73f9649a4ed2a30c950accf137dd in term 0.
I20260812 00:13:12.934310 1599 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b99a73f9649a4ed2a30c950accf137dd, c1855aa087f44db2a1ec2d27cefc02d4; no voters:
I20260812 00:13:12.934489 1737 raft_consensus.cc:2781] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:12.934546 1737 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:12.934568 1737 raft_consensus.cc:3037] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:12.935338 1737 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.935513 1737 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [CANDIDATE]: Term 1 election: Requested vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301)
I20260812 00:13:12.935662 1532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1370e2812b1d4402af1a860cebbe819e" candidate_uuid: "b99a73f9649a4ed2a30c950accf137dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d5733edead4de0897ed0741b8e12fc"
I20260812 00:13:12.935662 1381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1370e2812b1d4402af1a860cebbe819e" candidate_uuid: "b99a73f9649a4ed2a30c950accf137dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1855aa087f44db2a1ec2d27cefc02d4"
I20260812 00:13:12.935753 1532 raft_consensus.cc:3037] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:12.935776 1381 raft_consensus.cc:3037] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:12.936595 1532 raft_consensus.cc:2454] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b99a73f9649a4ed2a30c950accf137dd in term 1.
I20260812 00:13:12.936620 1381 raft_consensus.cc:2454] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b99a73f9649a4ed2a30c950accf137dd in term 1.
I20260812 00:13:12.936841 1599 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b99a73f9649a4ed2a30c950accf137dd, c1855aa087f44db2a1ec2d27cefc02d4; no voters:
I20260812 00:13:12.936967 1737 raft_consensus.cc:2781] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:12.937143 1737 raft_consensus.cc:686] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 1 LEADER]: Becoming Leader. State: Replica: b99a73f9649a4ed2a30c950accf137dd, State: Running, Role: LEADER
I20260812 00:13:12.937252 1737 consensus_queue.cc:237] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:12.938297 1228 catalog_manager.cc:5495] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd reported cstate change: term changed from 0 to 1, leader changed from <none> to b99a73f9649a4ed2a30c950accf137dd (127.16.99.195). New cstate: current_term: 1 leader_uuid: "b99a73f9649a4ed2a30c950accf137dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:12.971524 1731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.968873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.16.99.195:39351)
0812 00:13:12.968951 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:12.968953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:12.968963 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:13:12.969021 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:12.969022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:12.969042 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:12.969106 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:12.969221 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.969715 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.970553 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:12.970850 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:12.970913 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:12.971125 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:12.971248 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:12.971356 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:12.971374 (+ 18us) server_negotiation.cc:299] Negotiation successful
0812 00:13:12.971398 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 00:13:12.973769 1381 raft_consensus.cc:1261] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Refusing update from remote peer b99a73f9649a4ed2a30c950accf137dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:12.973795 1532 raft_consensus.cc:1261] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 1 FOLLOWER]: Refusing update from remote peer b99a73f9649a4ed2a30c950accf137dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:12.974169 1739 consensus_queue.cc:1035] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:12.974424 1737 consensus_queue.cc:1035] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:12.977308 1744 mvcc.cc:205] Tried to move back new op lower bound from 7317477756817260544 to 7317477756672057344. Current Snapshot: MvccSnapshot[applied={T|T < 7317477756817260544}]
I20260812 00:13:12.978008 1742 mvcc.cc:205] Tried to move back new op lower bound from 7317477756817260544 to 7317477756672057344. Current Snapshot: MvccSnapshot[applied={T|T < 7317477756817260544}]
I20260812 00:13:12.980712 1227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:12.980821 1227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:12.981957 1227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Sending DeleteTablet for 3 replicas of tablet 1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:12.982363 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet 1370e2812b1d4402af1a860cebbe819e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:12 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:51964
I20260812 00:13:12.982378 1512 tablet_service.cc:1514] Processing DeleteTablet for tablet 1370e2812b1d4402af1a860cebbe819e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:12 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:44592
I20260812 00:13:12.982491 1441 tablet_replica.cc:330] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: stopping tablet replica
I20260812 00:13:12.982553 1441 raft_consensus.cc:2229] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:12.982602 1441 raft_consensus.cc:2258] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:12.982746 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 1370e2812b1d4402af1a860cebbe819e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:12 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:34628
I20260812 00:13:12.982822 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 1268
I20260812 00:13:12.983826 1751 tablet_replica.cc:330] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: stopping tablet replica
I20260812 00:13:12.983985 1751 raft_consensus.cc:2229] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:12.984154 1751 raft_consensus.cc:2258] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:12.984910 1751 ts_tablet_manager.cc:1905] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:12.987293 1751 ts_tablet_manager.cc:1918] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:12.987402 1751 log.cc:1196] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:12.987747 1751 ts_tablet_manager.cc:1939] T 1370e2812b1d4402af1a860cebbe819e P 40d5733edead4de0897ed0741b8e12fc: Deleting consensus metadata
I20260812 00:13:12.988394 1213 catalog_manager.cc:4841] TS 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663): tablet 1370e2812b1d4402af1a860cebbe819e (table test-table [id=e909f4bd7e384d45b96aed13d5f4abd2]) successfully deleted
W20260812 00:13:12.992861 1212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:34301 (error 108)
I20260812 00:13:12.993141 1752 tablet_replica.cc:330] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: stopping tablet replica
I20260812 00:13:12.993345 1752 raft_consensus.cc:2229] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:13:12.993394 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 1438
I20260812 00:13:12.993517 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:12.993367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:34301 (local address 127.0.0.1:51968)
0812 00:13:12.993428 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:12.993471 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:34301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:13:12.993698 1752 raft_consensus.cc:2258] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 00:13:12.993667 1212 catalog_manager.cc:4584] TS c1855aa087f44db2a1ec2d27cefc02d4 (127.16.99.193:34301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1370e2812b1d4402af1a860cebbe819e: Network error: Client connection negotiation failed: client connection to 127.16.99.193:34301: connect: Connection refused (error 111)
I20260812 00:13:12.994735 1752 ts_tablet_manager.cc:1905] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:12.996637 1752 ts_tablet_manager.cc:1918] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:12.996804 1752 log.cc:1196] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:12.997186 1752 ts_tablet_manager.cc:1939] T 1370e2812b1d4402af1a860cebbe819e P b99a73f9649a4ed2a30c950accf137dd: Deleting consensus metadata
I20260812 00:13:12.997726 1213 catalog_manager.cc:4841] TS b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): tablet 1370e2812b1d4402af1a860cebbe819e (table test-table [id=e909f4bd7e384d45b96aed13d5f4abd2]) successfully deleted
I20260812 00:13:13.001606 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 1583
I20260812 00:13:13.009030 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 1190
I20260812 00:13:33.016151 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37271
--webserver_interface=127.16.99.254
--webserver_port=33353
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:33.107574 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:33.107818 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:33.107844 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:33.109123 1754 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 00:13:33.109174 1754 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 00:13:33.109187 1754 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 00:13:33.109200 1754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:33.109220 1754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:33.109233 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:33.109246 1754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:33.109287 1754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:33.110913 1754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=33353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:33.111210 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:33.111469 1754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:33.117470 1763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:33.117492 1762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:33.117515 1754 server_base.cc:1034] running on GCE node
W20260812 00:13:33.117492 1765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:33.118031 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:33.118309 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:33.119509 1754 hybrid_clock.cc:648] HybridClock initialized: now 1786493613119490 us; error 42 us; skew 500 ppm
Aug 12 00:13:33 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:33.122589 1754 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:13:33.123150 1754 webserver.cc:460] Webserver started at http://127.16.99.254:33353/ using document root <none> and password file <none>
I20260812 00:13:33.123440 1754 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:33.123499 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:33.125044 1754 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:13:33.125985 1773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.126308 1754 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:33.126480 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "57e08f90af1c4069a60f4c36a30ac556"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "f34cb93ff3e90c738a6aaef11dc7d4d3"
server_key_iv: "ae34b117b83f42189c7a03321cece8e8"
server_key_version: "encryptionkey@0"
I20260812 00:13:33.126889 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:33.144783 1754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:33.147454 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:33.147748 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:33.152201 1754 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:37271
I20260812 00:13:33.152233 1829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:37271 every 8 connection(s)
I20260812 00:13:33.152681 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:33.153637 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 1754
I20260812 00:13:33.154084 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:34301
--local_ip_for_outbound_sockets=127.16.99.193
--tserver_master_addrs=127.16.99.254:37271
--webserver_port=44291
--webserver_interface=127.16.99.193
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:13:33.155025 1830 sys_catalog.cc:263] Verifying existing consensus state
I20260812 00:13:33.156239 1830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Bootstrap starting.
I20260812 00:13:33.165760 1830 log.cc:824] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:33.168283 1830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:13:33.168642 1830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Bootstrap complete.
I20260812 00:13:33.171197 1830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:33.171482 1830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e08f90af1c4069a60f4c36a30ac556, State: Initialized, Role: FOLLOWER
I20260812 00:13:33.171703 1830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:33.171813 1830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:33.171869 1830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:33.171921 1830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 2 FOLLOWER]: Advancing to term 3
I20260812 00:13:33.173120 1830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:33.173257 1830 leader_election.cc:302] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57e08f90af1c4069a60f4c36a30ac556; no voters:
I20260812 00:13:33.173486 1830 leader_election.cc:288] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 00:13:33.173528 1834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 00:13:33.173658 1834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [term 3 LEADER]: Becoming Leader. State: Replica: 57e08f90af1c4069a60f4c36a30ac556, State: Running, Role: LEADER
I20260812 00:13:33.173785 1830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:33.173835 1834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } }
I20260812 00:13:33.174136 1834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57e08f90af1c4069a60f4c36a30ac556. Latest consensus state: current_term: 3 leader_uuid: "57e08f90af1c4069a60f4c36a30ac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } } }
I20260812 00:13:33.174220 1834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:33.174144 1836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "57e08f90af1c4069a60f4c36a30ac556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e08f90af1c4069a60f4c36a30ac556" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37271 } } }
I20260812 00:13:33.174435 1836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:33.174474 1840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:33.175527 1840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:33.175695 1840 catalog_manager.cc:1259] Loaded cluster ID: 3b39026ff976421eab4e3252ab4545ce
I20260812 00:13:33.176024 1840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:33.176893 1840 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:33.177094 1840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Loaded TSK: 0
I20260812 00:13:33.177155 1840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57e08f90af1c4069a60f4c36a30ac556: Loaded TSK: 1
F20260812 00:13:33.177388 1840 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f38c85749ad google::LogMessage::Fail() at ??:0
@ 0x7f38c857788a google::LogMessage::SendToLog() at ??:0
@ 0x7f38c85744de google::LogMessage::Flush() at ??:0
@ 0x7f38c8575a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f38cace1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f38cacded41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f38d04d21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f38d04d248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f38d04d2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f38d0498c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f38d04a2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f38ca4420d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f38ca437954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f38cb29b6db start_thread at ??:0
@ 0x7f38c707261f clone at ??:0
W20260812 00:13:33.259601 1832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:33.259891 1832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:33.259941 1832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:33.261356 1832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:33.261443 1832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:33.261495 1832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:33.263401 1832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:34301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=44291
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:33.263733 1832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:33.264005 1832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:33.269675 1876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:33.269737 1877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:33.269860 1832 server_base.cc:1034] running on GCE node
W20260812 00:13:33.269742 1879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:33.270385 1832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:33.270668 1832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:33.271850 1832 hybrid_clock.cc:648] HybridClock initialized: now 1786493613271821 us; error 44 us; skew 500 ppm
Aug 12 00:13:33 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:33.274860 1832 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:13:33.275511 1832 webserver.cc:460] Webserver started at http://127.16.99.193:44291/ using document root <none> and password file <none>
I20260812 00:13:33.275791 1832 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:33.275879 1832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:33.277874 1832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:33.278831 1888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.279078 1832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:13:33.279197 1832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "c1855aa087f44db2a1ec2d27cefc02d4"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "bae3d4881b05bec936873b99dd745301"
server_key_iv: "49dacd03d79e3082cbbcc6f338ab1340"
server_key_version: "encryptionkey@0"
I20260812 00:13:33.279691 1832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:33.299620 1832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:33.303424 1832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:33.303612 1832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:33.303939 1832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:33.304550 1895 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 00:13:33.306284 1832 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 00:13:33.306401 1832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 00:13:33.306443 1832 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 00:13:33.307392 1832 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 00:13:33.307437 1832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 00:13:33.307539 1895 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap starting.
I20260812 00:13:33.315259 1832 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:34301
I20260812 00:13:33.315514 2014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:34301 every 8 connection(s)
I20260812 00:13:33.315763 1832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:33.319808 1895 log.cc:824] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:33.321269 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 1832
I20260812 00:13:33.321630 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:35663
--local_ip_for_outbound_sockets=127.16.99.194
--tserver_master_addrs=127.16.99.254:37271
--webserver_port=39435
--webserver_interface=127.16.99.194
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:13:33.323117 1895 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 00:13:33.323453 1895 tablet_bootstrap.cc:492] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Bootstrap complete.
I20260812 00:13:33.324757 1895 ts_tablet_manager.cc:1397] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260812 00:13:33.325906 1895 raft_consensus.cc:348] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:33.326184 1895 raft_consensus.cc:729] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1855aa087f44db2a1ec2d27cefc02d4, State: Initialized, Role: FOLLOWER
I20260812 00:13:33.326408 1895 consensus_queue.cc:260] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:33.326742 1895 ts_tablet_manager.cc:1428] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
W20260812 00:13:33.415139 2017 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:13:33.316625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:41291)
0812 00:13:33.316886 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.316944 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:33.317792 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:33.414920 (+ 97128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
W20260812 00:13:33.415570 2015 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:37271 (0 consecutive failures): Network error: Failed to ping master at 127.16.99.254:37271: Client connection negotiation failed: client connection to 127.16.99.254:37271: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 00:13:33.415946 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.415819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:33819)
0812 00:13:33.415875 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.415923 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 00:13:33.416399 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.416304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:54685)
0812 00:13:33.416355 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.416381 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 00:13:33.416549 2015 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:13:33.418747 2019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:33.418993 2019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:33.419039 2019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:33.420390 2019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:33.420466 2019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:33.420516 2019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:33.422237 2019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:35663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=39435
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:33.422557 2019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:33.422786 2019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:33.428637 2026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:33.428640 2029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:33.428715 2027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:33.429184 2019 server_base.cc:1034] running on GCE node
I20260812 00:13:33.429605 2019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:33.429953 2019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:33.431164 2019 hybrid_clock.cc:648] HybridClock initialized: now 1786493613431133 us; error 43 us; skew 500 ppm
Aug 12 00:13:33 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:33.434420 2019 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:13:33.434974 2019 webserver.cc:460] Webserver started at http://127.16.99.194:39435/ using document root <none> and password file <none>
I20260812 00:13:33.435281 2019 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:33.435388 2019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:33.437129 2019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:33.438194 2037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.438488 2019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:33.438607 2019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "40d5733edead4de0897ed0741b8e12fc"
format_stamp: "Formatted at 2026-08-12 00:12:51 on dist-test-slave-zpfg"
server_key: "e652d4a1965001f418c02ef5548b48fe"
server_key_iv: "c0a11467db348559f1f0c056e3deb130"
server_key_version: "encryptionkey@0"
I20260812 00:13:33.439049 2019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:33.457983 2019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:33.461189 2019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:33.461387 2019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:33.461663 2019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:33.462031 2019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:33.462092 2019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.462136 2019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:33.462193 2019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.469166 2019 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:35663
I20260812 00:13:33.469270 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:35663 every 8 connection(s)
I20260812 00:13:33.469673 2019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:13:33.470913 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 2019
I20260812 00:13:33.471133 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.470539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:37229)
0812 00:13:33.470781 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.470901 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260812 00:13:33.471442 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:39351
--local_ip_for_outbound_sockets=127.16.99.195
--tserver_master_addrs=127.16.99.254:37271
--webserver_port=45157
--webserver_interface=127.16.99.195
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:33.471757 2163 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:37271 (0 consecutive failures): Network error: Failed to ping master at 127.16.99.254:37271: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
I20260812 00:13:33.472107 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.471975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:48601)
0812 00:13:33.472035 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.472073 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:13:33.473374 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.472384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:55715)
0812 00:13:33.473302 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.473338 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":896}
W20260812 00:13:33.473500 2163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 00:13:33.559859 2166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:33.560133 2166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:33.560186 2166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:33.561457 2166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:33.561537 2166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:33.561585 2166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:33.563321 2166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:39351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=45157
--tserver_master_addrs=127.16.99.254:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:33.563647 2166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:33.563907 2166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:33.569756 2173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:33.569829 2172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:33.569756 2176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:33.569948 2166 server_base.cc:1034] running on GCE node
I20260812 00:13:33.570246 2166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:33.570539 2166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:33.571702 2166 hybrid_clock.cc:648] HybridClock initialized: now 1786493613571700 us; error 81 us; skew 500 ppm
Aug 12 00:13:33 dist-test-slave-zpfg krb5kdc[606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:33.574862 2166 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:13:33.575492 2166 webserver.cc:460] Webserver started at http://127.16.99.195:45157/ using document root <none> and password file <none>
I20260812 00:13:33.575773 2166 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:33.575848 2166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:33.577435 2166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:33.578382 2184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.578725 2166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:33.578884 2166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "b99a73f9649a4ed2a30c950accf137dd"
format_stamp: "Formatted at 2026-08-12 00:12:52 on dist-test-slave-zpfg"
server_key: "ceb536b6fb4d6e86197583ac303b50ad"
server_key_iv: "7dc8dabde636249584670bfb827213fb"
server_key_version: "encryptionkey@0"
I20260812 00:13:33.579329 2166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:33.593132 2166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:33.595979 2166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:33.596190 2166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:33.596503 2166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:33.596858 2166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:33.596916 2166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.596959 2166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:33.596988 2166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:33.603937 2166 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:39351
I20260812 00:13:33.604043 2304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:39351 every 8 connection(s)
I20260812 00:13:33.604555 2166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:13:33.605886 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.605424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:48499)
0812 00:13:33.605601 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.605704 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
W20260812 00:13:33.606521 2305 heartbeater.cc:644] Failed to heartbeat to 127.16.99.254:37271 (0 consecutive failures): Network error: Failed to ping master at 127.16.99.254:37271: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
I20260812 00:13:33.606853 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.606739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:43615)
0812 00:13:33.606783 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.606827 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260812 00:13:33.607374 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:33.607257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:43937)
0812 00:13:33.607322 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:33.607353 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 00:13:33.607594 2305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 00:13:33.608968 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 2166
I20260812 00:13:34.417284 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:34.416940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:40353)
0812 00:13:34.417178 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:34.417219 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260812 00:13:34.474327 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:34.473917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:57759)
0812 00:13:34.474077 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:34.474268 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260812 00:13:34.608270 2313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:34.608019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:50159)
0812 00:13:34.608165 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:34.608211 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260812 00:13:34.841859 2314 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:34.842056 2314 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:34.842489 2314 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
Aug 12 00:13:34 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.194@KRBTEST.COM
Aug 12 00:13:34 dist-test-slave-zpfg krb5kdc[606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493613, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.195@KRBTEST.COM
I20260812 00:13:34.853289 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:34.842678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:35663 (local address 127.16.99.193:44061)
0812 00:13:34.842741 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:34.842759 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:34.842842 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:34.843993 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:34.843996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:34.844378 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:34.844563 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:34.844569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.845668 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.845669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:34.846580 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:34.846582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.847179 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.847180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:34.847291 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:34.847974 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:34.847978 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:34.849444 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:34.851626 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:34.851633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:34.851635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:34.851945 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:34.852214 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:34.852215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:34.852216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:34.852349 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:34.852756 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:34.852764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:34.852892 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 00:13:34.853004 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 00:13:34.853174 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":29}
I20260812 00:13:34.853300 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:34.842795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:44061 (local address 127.16.99.194:35663)
0812 00:13:34.842989 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:34.843002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:34.843041 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:13:34.843754 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:34.843756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:34.843804 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:34.843921 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:34.844707 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.845556 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.846687 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.847077 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.847202 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:34.849561 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:34.849565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:34.849566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:34.849578 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:34.851523 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:34.852035 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:34.852037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:34.852038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:34.852123 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:34.852428 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:34.852429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:34.852430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:34.852666 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:34.852755 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:34.853054 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 00:13:34.853177 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260812 00:13:34.853570 2317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:34.842904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:40659 (local address 127.16.99.195:39351)
0812 00:13:34.843057 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:34.843068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:34.843206 (+ 138us) server_negotiation.cc:407] Connection header received
0812 00:13:34.843844 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:34.843847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:34.843922 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:34.844056 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:34.844687 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.845574 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.846666 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.847070 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.847199 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:34.849907 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:34.849911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:34.849911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:34.849935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:34.852093 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:34.852451 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:34.852454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:34.852455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:34.852541 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:34.852812 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:34.852813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:34.852813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:34.853035 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:34.853136 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:34.853320 (+ 184us) server_negotiation.cc:299] Negotiation successful
0812 00:13:34.853445 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260812 00:13:34.853468 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:34.842818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:39351 (local address 127.16.99.193:40659)
0812 00:13:34.842929 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:34.842940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:34.842977 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:34.844044 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:34.844045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:34.844376 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:34.844563 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:34.844569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.845701 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.845701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:34.846582 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:34.846584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:34.847162 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:34.847163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:34.847289 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:34.847911 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:34.847919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:34.849801 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:34.852185 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:34.852185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:34.852186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:34.852357 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:34.852671 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:34.852672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:34.852672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:34.852736 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:34.853117 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:34.853118 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:34.853163 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 00:13:34.853259 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 00:13:34.853361 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":35,"thread_start_us":58,"threads_started":1}
W20260812 00:13:34.854797 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
W20260812 00:13:34.854876 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:34.854894 1890 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1855aa087f44db2a1ec2d27cefc02d4; no voters: 40d5733edead4de0897ed0741b8e12fc, b99a73f9649a4ed2a30c950accf137dd
I20260812 00:13:34.855058 2314 raft_consensus.cc:2726] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:13:35.418216 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:35.417937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:57593)
0812 00:13:35.418115 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:35.418167 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260812 00:13:35.475324 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:35.474992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:55569)
0812 00:13:35.475226 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:35.475274 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":71,"threads_started":1}
I20260812 00:13:35.609097 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:35.608854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:60375)
0812 00:13:35.608994 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:35.609040 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260812 00:13:36.419063 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:36.418808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:42453)
0812 00:13:36.418967 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:36.419013 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260812 00:13:36.476207 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:36.475896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:44325)
0812 00:13:36.476076 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:36.476146 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260812 00:13:36.609958 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:36.609702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:36703)
0812 00:13:36.609862 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:36.609909 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260812 00:13:36.990139 2326 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:36.990278 2326 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:36.990530 2326 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
W20260812 00:13:36.991039 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
W20260812 00:13:36.991130 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:36.991151 1890 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1855aa087f44db2a1ec2d27cefc02d4; no voters: 40d5733edead4de0897ed0741b8e12fc, b99a73f9649a4ed2a30c950accf137dd
I20260812 00:13:36.991230 2326 raft_consensus.cc:2726] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:13:37.420013 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:37.419732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:47311)
0812 00:13:37.419913 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:37.419963 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260812 00:13:37.477216 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:37.476950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:44197)
0812 00:13:37.477099 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:37.477146 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260812 00:13:37.610836 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:37.610557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:43277)
0812 00:13:37.610741 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:37.610786 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":118,"threads_started":1}
I20260812 00:13:38.420944 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:38.420668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:39559)
0812 00:13:38.420843 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:38.420892 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260812 00:13:38.478101 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:38.477831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:42731)
0812 00:13:38.478002 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:38.478051 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260812 00:13:38.611651 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:38.611410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:37011)
0812 00:13:38.611557 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:38.611605 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260812 00:13:39.421965 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:39.421644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:35677)
0812 00:13:39.421857 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:39.421914 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
I20260812 00:13:39.478981 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:39.478717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:51799)
0812 00:13:39.478878 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:39.478926 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260812 00:13:39.612574 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:39.612287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:43681)
0812 00:13:39.612475 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:39.612524 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20260812 00:13:40.070871 2336 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:40.070988 2336 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:40.071295 2336 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
W20260812 00:13:40.071810 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
W20260812 00:13:40.071929 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:40.071972 1890 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1855aa087f44db2a1ec2d27cefc02d4; no voters: 40d5733edead4de0897ed0741b8e12fc, b99a73f9649a4ed2a30c950accf137dd
I20260812 00:13:40.072086 2336 raft_consensus.cc:2726] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:13:40.422955 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:40.422699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:45557)
0812 00:13:40.422862 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:40.422906 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260812 00:13:40.479765 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:40.479520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:48171)
0812 00:13:40.479674 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:40.479717 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260812 00:13:40.613355 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:40.613127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:37301)
0812 00:13:40.613271 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:40.613308 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260812 00:13:41.423969 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:41.423697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:35433)
0812 00:13:41.423867 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:41.423919 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260812 00:13:41.480700 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:41.480419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:42879)
0812 00:13:41.480590 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:41.480630 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260812 00:13:41.614151 2344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:41.613857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:54743)
0812 00:13:41.614041 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:41.614092 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260812 00:13:42.219444 2345 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:42.219585 2345 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:42.219833 2345 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
W20260812 00:13:42.220371 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
W20260812 00:13:42.220497 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:42.220536 1890 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1855aa087f44db2a1ec2d27cefc02d4; no voters: 40d5733edead4de0897ed0741b8e12fc, b99a73f9649a4ed2a30c950accf137dd
I20260812 00:13:42.220639 2345 raft_consensus.cc:2726] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:13:42.424959 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:42.424620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:51663)
0812 00:13:42.424854 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:42.424907 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":127,"threads_started":1}
I20260812 00:13:42.481642 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:42.481362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:55535)
0812 00:13:42.481545 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:42.481591 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260812 00:13:42.615000 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:42.614763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:32887)
0812 00:13:42.614901 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:42.614948 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260812 00:13:43.425792 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:43.425543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:54873)
0812 00:13:43.425689 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:43.425740 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260812 00:13:43.482759 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:43.482465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:53841)
0812 00:13:43.482625 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:43.482707 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260812 00:13:43.615886 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:43.615626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:52357)
0812 00:13:43.615792 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:43.615836 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260812 00:13:44.328872 2353 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:44.328970 2353 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:44.329181 2353 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
W20260812 00:13:44.329576 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
W20260812 00:13:44.329679 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:44.329705 1890 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1855aa087f44db2a1ec2d27cefc02d4; no voters: 40d5733edead4de0897ed0741b8e12fc, b99a73f9649a4ed2a30c950accf137dd
I20260812 00:13:44.329792 2353 raft_consensus.cc:2726] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:13:44.426813 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:44.426571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:49989)
0812 00:13:44.426715 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:44.426763 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260812 00:13:44.483567 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:44.483321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:33885)
0812 00:13:44.483464 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:44.483515 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260812 00:13:44.616845 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:44.616533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:59329)
0812 00:13:44.616738 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:44.616788 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
I20260812 00:13:45.427718 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:45.427424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:35127)
0812 00:13:45.427603 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:45.427654 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260812 00:13:45.484514 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:45.484262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:50337)
0812 00:13:45.484414 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:45.484462 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260812 00:13:45.617760 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:45.617407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:52737)
0812 00:13:45.617598 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:45.617700 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260812 00:13:46.428668 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:46.428408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:58311)
0812 00:13:46.428572 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:46.428617 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260812 00:13:46.485402 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:46.485167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:35373)
0812 00:13:46.485306 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:46.485351 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260812 00:13:46.618503 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:46.618193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:47217)
0812 00:13:46.618402 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:46.618453 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260812 00:13:47.429538 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:47.429290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:58517)
0812 00:13:47.429440 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:47.429483 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260812 00:13:47.486447 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:47.485999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:50197)
0812 00:13:47.486321 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:47.486394 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":74,"threads_started":1}
I20260812 00:13:47.619472 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:47.619075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:33191)
0812 00:13:47.619357 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:47.619420 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260812 00:13:48.430536 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:48.430193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.193:42031)
0812 00:13:48.430444 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:48.430489 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
I20260812 00:13:48.487310 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:48.487045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.194:38253)
0812 00:13:48.487204 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:48.487255 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260812 00:13:48.509410 2368 raft_consensus.cc:482] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:48.509537 2368 raft_consensus.cc:504] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d5733edead4de0897ed0741b8e12fc" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 35663 } } peers { permanent_uuid: "b99a73f9649a4ed2a30c950accf137dd" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39351 } } peers { permanent_uuid: "c1855aa087f44db2a1ec2d27cefc02d4" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 34301 } }
I20260812 00:13:48.509789 2368 leader_election.cc:288] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663), b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351)
W20260812 00:13:48.510313 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40d5733edead4de0897ed0741b8e12fc (127.16.99.194:35663): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
W20260812 00:13:48.510429 1890 leader_election.cc:341] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b99a73f9649a4ed2a30c950accf137dd (127.16.99.195:39351): Not found: Tablet not found: 1370e2812b1d4402af1a860cebbe819e
I20260812 00:13:48.510469 1890 leader_election.cc:302] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1855aa087f44db2a1ec2d27cefc02d4; no voters: 40d5733edead4de0897ed0741b8e12fc, b99a73f9649a4ed2a30c950accf137dd
I20260812 00:13:48.510571 2368 raft_consensus.cc:2726] T 1370e2812b1d4402af1a860cebbe819e P c1855aa087f44db2a1ec2d27cefc02d4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 00:13:48.615422 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 1832
I20260812 00:13:48.620404 2369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:48.620160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37271 (local address 127.16.99.195:38475)
0812 00:13:48.620296 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:48.620341 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.254:37271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260812 00:13:48.623067 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 2019
I20260812 00:13:48.629251 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 2166
2026-08-12T00:13:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58604 ms)
[----------] 30 tests from SecurityITest (247031 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:13:48 dist-test-slave-zpfg krb5kdc[2372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:13:48 dist-test-slave-zpfg krb5kdc[2372](info): set up 2 sockets
Aug 12 00:13:48 dist-test-slave-zpfg krb5kdc[2372](info): commencing operation
krb5kdc: starting...
W20260812 00:13:49.800501 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:13:49 dist-test-slave-zpfg krb5kdc[2372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:13:49.941684 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40475
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:40475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:50.033237 2385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:50.033510 2385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:50.033537 2385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:50.034943 2385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:50.034996 2385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:50.035012 2385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:50.035027 2385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:50.035040 2385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:50.037281 2385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:40475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:50.037566 2385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:50.037786 2385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:50.043828 2394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:50.043824 2391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:50.043859 2385 server_base.cc:1034] running on GCE node
W20260812 00:13:50.043820 2390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:50.044538 2385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:50.044880 2385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:50.046083 2385 hybrid_clock.cc:648] HybridClock initialized: now 1786493630046062 us; error 49 us; skew 500 ppm
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493630, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:50.049160 2385 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:13:50.049765 2385 webserver.cc:460] Webserver started at http://127.16.99.254:33561/ using document root <none> and password file <none>
I20260812 00:13:50.050037 2385 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:50.050125 2385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:50.050298 2385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:50.051607 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "5b69b936bc4845ac9ab183b06a537ab4"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "1b32f54f64a3087f374d13be452ad735"
server_key_iv: "4e211244689c5cf408e1a8b7a5bdbc9c"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.052042 2385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "5b69b936bc4845ac9ab183b06a537ab4"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "1b32f54f64a3087f374d13be452ad735"
server_key_iv: "4e211244689c5cf408e1a8b7a5bdbc9c"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.053679 2385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:13:50.054783 2402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.055004 2385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:50.055131 2385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "5b69b936bc4845ac9ab183b06a537ab4"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "1b32f54f64a3087f374d13be452ad735"
server_key_iv: "4e211244689c5cf408e1a8b7a5bdbc9c"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.055326 2385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:50.077925 2385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:50.081849 2385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:50.082034 2385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:50.087749 2385 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:40475
I20260812 00:13:50.088276 2385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:50.088517 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 2385
I20260812 00:13:50.088654 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:13:50.088922 16783 external_mini_cluster.cc:1428] Setting key 3118df654e8922551d6739946f00fd1f
I20260812 00:13:50.089272 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:40475 every 8 connection(s)
I20260812 00:13:50.089560 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:50.097545 2463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4: Bootstrap starting.
I20260812 00:13:50.098294 2463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:50.098650 2463 log.cc:824] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4: Log is configured to *not* fsync() on all Append() calls
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:50.099447 2463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4: No bootstrap required, opened a new log
I20260812 00:13:50.101257 2463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69b936bc4845ac9ab183b06a537ab4" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40475 } }
I20260812 00:13:50.101440 2463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:50.101492 2463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b69b936bc4845ac9ab183b06a537ab4, State: Initialized, Role: FOLLOWER
I20260812 00:13:50.101660 2463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69b936bc4845ac9ab183b06a537ab4" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40475 } }
I20260812 00:13:50.101755 2463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:50.101810 2463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:50.101858 2463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:50.102586 2463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69b936bc4845ac9ab183b06a537ab4" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40475 } }
I20260812 00:13:50.102715 2463 leader_election.cc:302] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b69b936bc4845ac9ab183b06a537ab4; no voters:
I20260812 00:13:50.102905 2463 leader_election.cc:288] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:50.103003 2468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:50.103209 2468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 5b69b936bc4845ac9ab183b06a537ab4, State: Running, Role: LEADER
I20260812 00:13:50.103214 2463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:50.103420 2468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69b936bc4845ac9ab183b06a537ab4" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40475 } }
I20260812 00:13:50.104128 2470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b69b936bc4845ac9ab183b06a537ab4. Latest consensus state: current_term: 1 leader_uuid: "5b69b936bc4845ac9ab183b06a537ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69b936bc4845ac9ab183b06a537ab4" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40475 } } }
I20260812 00:13:50.104226 2470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:50.105005 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.089568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.16.99.254:40475)
0812 00:13:50.089873 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.089884 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.089935 (+ 51us) server_negotiation.cc:407] Connection header received
0812 00:13:50.090676 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.090689 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.090769 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.090969 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:50.091932 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.093726 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.095071 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.095809 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.095932 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.099439 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:50.099445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:50.099445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:50.099460 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:50.102534 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.102990 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.103001 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.103001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.103157 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.103459 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.103460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.103460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.103830 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:50.103969 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.104575 (+ 606us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.104778 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":85,"threads_started":1}
I20260812 00:13:50.105072 2469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b69b936bc4845ac9ab183b06a537ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b69b936bc4845ac9ab183b06a537ab4" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40475 } } }
I20260812 00:13:50.105162 2469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4 [sys.catalog]: This master's current role is: LEADER
W20260812 00:13:50.105665 2483 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:13:50.105751 2483 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:13:50.105834 2477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:50.106050 2477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:50.107858 2477 catalog_manager.cc:1347] Generated new cluster ID: 3082b0030c21402bbb8bd7b49bc9d0ac
I20260812 00:13:50.107910 2477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:50.116211 2477 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:50.116813 2477 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:50.122519 2477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4: Generated new TSK 0
I20260812 00:13:50.122727 2477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:13:50.162292 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40475
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:50.280633 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:50.280872 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:50.280901 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:50.282097 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:50.282371 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:50.282415 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:50.284320 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:50.284587 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:50.284806 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:50.290622 2498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:50.290710 2496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:50.290642 2495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:50.291271 2489 server_base.cc:1034] running on GCE node
I20260812 00:13:50.291520 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:50.291808 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:50.293006 2489 hybrid_clock.cc:648] HybridClock initialized: now 1786493630293003 us; error 53 us; skew 500 ppm
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493630, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:50.296175 2489 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:13:50.296774 2489 webserver.cc:460] Webserver started at http://127.16.99.193:44721/ using document root <none> and password file <none>
I20260812 00:13:50.297046 2489 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:50.297120 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:50.297330 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:50.298566 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "07f6b9559b84461894a88646c3761be6"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "0d383f7bf07a770be2604ab541a9c014"
server_key_iv: "099b3ff8916a389e9f1ead646ff44adf"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.298966 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "07f6b9559b84461894a88646c3761be6"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "0d383f7bf07a770be2604ab541a9c014"
server_key_iv: "099b3ff8916a389e9f1ead646ff44adf"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.300644 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:50.301570 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.301896 2489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:50.302027 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "07f6b9559b84461894a88646c3761be6"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "0d383f7bf07a770be2604ab541a9c014"
server_key_iv: "099b3ff8916a389e9f1ead646ff44adf"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.302126 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:50.316009 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:50.319168 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:50.319383 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:50.319670 2489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:50.320034 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:50.320101 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.320138 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:50.320171 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.328186 2489 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:46395
I20260812 00:13:50.328280 2628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:46395 every 8 connection(s)
I20260812 00:13:50.328694 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:50.330667 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 2489
I20260812 00:13:50.330845 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:13:50.331099 16783 external_mini_cluster.cc:1428] Setting key 27121551da505d21c84a609f6b83ea3e
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493630, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:50.342272 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.329901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:54237 (local address 127.16.99.254:40475)
0812 00:13:50.330002 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.330005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.330199 (+ 194us) server_negotiation.cc:407] Connection header received
0812 00:13:50.331260 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.331262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.331283 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.331381 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:50.333028 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.333688 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.335052 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.335307 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.335414 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.338671 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:50.338673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:50.338673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:50.338683 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:50.340505 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.341088 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.341089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.341089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.341157 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.341482 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.341483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.341484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.341628 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:50.341713 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.342048 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.342150 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 00:13:50.342499 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.329752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40475 (local address 127.16.99.193:54237)
0812 00:13:50.330043 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.330100 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.330984 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.331490 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.331495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.332473 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:50.332846 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.332856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.333823 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.333824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.334932 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.334935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.335434 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.335435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.335616 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.336258 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:50.336270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:50.338542 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:50.340619 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:50.340631 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:50.340634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:50.340988 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:50.341254 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:50.341255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:50.341256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:50.341401 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:50.341703 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:50.341711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:50.341890 (+ 179us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.342010 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.342209 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":131,"threads_started":1}
I20260812 00:13:50.343099 2631 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40475
I20260812 00:13:50.343321 2631 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
I20260812 00:13:50.343609 2631 heartbeater.cc:511] Master 127.16.99.254:40475 requested a full tablet report, sending...
I20260812 00:13:50.344405 2416 ts_manager.cc:194] Registered new tserver with Master: 07f6b9559b84461894a88646c3761be6 (127.16.99.193:46395)
I20260812 00:13:50.345103 2416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:54237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:13:50.366576 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40475
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:50.462456 2637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:50.462759 2637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:50.462810 2637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:50.464257 2637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:50.464354 2637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:50.464417 2637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:50.466416 2637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:50.466770 2637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:50.467046 2637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:50.472646 2643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:50.472622 2645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:50.472599 2642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:50.472848 2637 server_base.cc:1034] running on GCE node
I20260812 00:13:50.473361 2637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:50.473820 2637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:50.475049 2637 hybrid_clock.cc:648] HybridClock initialized: now 1786493630475027 us; error 37 us; skew 500 ppm
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493630, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:50.478101 2637 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:13:50.478660 2637 webserver.cc:460] Webserver started at http://127.16.99.194:35881/ using document root <none> and password file <none>
I20260812 00:13:50.479076 2637 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:50.479259 2637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:50.479440 2637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:50.480670 2637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "416fb4c81de84f4191c4048609b26541"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "6bdd778d7c5d1c4c9acd770c4029bb8e"
server_key_iv: "0dea80c173ad74d6ae72e3ec6e5e22a5"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.481113 2637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "416fb4c81de84f4191c4048609b26541"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "6bdd778d7c5d1c4c9acd770c4029bb8e"
server_key_iv: "0dea80c173ad74d6ae72e3ec6e5e22a5"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.482733 2637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260812 00:13:50.483762 2653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.484037 2637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:50.484156 2637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "416fb4c81de84f4191c4048609b26541"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "6bdd778d7c5d1c4c9acd770c4029bb8e"
server_key_iv: "0dea80c173ad74d6ae72e3ec6e5e22a5"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.484243 2637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:50.504911 2637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:50.507551 2637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:50.507781 2637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:50.508065 2637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:50.508445 2637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:50.508502 2637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.508538 2637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:50.508566 2637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.515168 2637 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:40875
I20260812 00:13:50.515233 2772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:40875 every 8 connection(s)
I20260812 00:13:50.515585 2637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493630, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:50.524077 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 2637
I20260812 00:13:50.524214 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:13:50.524654 16783 external_mini_cluster.cc:1428] Setting key 41f75da756773666b0e75d266a0391a4
I20260812 00:13:50.526780 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.516471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:36895 (local address 127.16.99.254:40475)
0812 00:13:50.516563 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.516566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.516755 (+ 189us) server_negotiation.cc:407] Connection header received
0812 00:13:50.517304 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.517306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.517317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.517372 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:50.518405 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.518956 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.520012 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.520229 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.520316 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.522847 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:50.522849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:50.522849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:50.522856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:50.524890 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.525494 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.525496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.525496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.525572 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.525916 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.525916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.525917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.526037 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:50.526096 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.526587 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.526709 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 00:13:50.527029 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.516440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40475 (local address 127.16.99.194:36895)
0812 00:13:50.516620 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.516666 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.517182 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.517464 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.517468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.518052 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:50.518272 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.518277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.519061 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.519063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.519906 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.519908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.520329 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.520335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.520471 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.521049 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:50.521058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:50.522719 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:50.525024 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:50.525034 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:50.525036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:50.525407 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:50.525660 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:50.525661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:50.525661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:50.525832 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:50.526107 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:50.526114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:50.526400 (+ 286us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.526530 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.526732 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260812 00:13:50.527510 2773 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40475
I20260812 00:13:50.527678 2773 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:50.527918 2773 heartbeater.cc:511] Master 127.16.99.254:40475 requested a full tablet report, sending...
I20260812 00:13:50.528391 2416 ts_manager.cc:194] Registered new tserver with Master: 416fb4c81de84f4191c4048609b26541 (127.16.99.194:40875)
I20260812 00:13:50.528949 2416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:36895
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:13:50.558463 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40475
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:34481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:50.651942 2779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:50.652319 2779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:50.652352 2779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:50.653648 2779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:50.653709 2779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:50.653739 2779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:50.655419 2779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:50.655700 2779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:50.655958 2779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:50.662186 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:50.662330 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:50.662180 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:50.662408 2779 server_base.cc:1034] running on GCE node
I20260812 00:13:50.662736 2779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:50.663003 2779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:50.664207 2779 hybrid_clock.cc:648] HybridClock initialized: now 1786493630664193 us; error 38 us; skew 500 ppm
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493630, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:50.667613 2779 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:13:50.668181 2779 webserver.cc:460] Webserver started at http://127.16.99.195:36617/ using document root <none> and password file <none>
I20260812 00:13:50.668442 2779 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:50.668514 2779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:50.668715 2779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:50.669848 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "78ca68890fca41d084ac33d482b77a5b"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "253e9e5a863d254d1cb9662cd292ea24"
server_key_iv: "7da2c6aa25ff2c49642b194d7196b7cc"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.670262 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "78ca68890fca41d084ac33d482b77a5b"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "253e9e5a863d254d1cb9662cd292ea24"
server_key_iv: "7da2c6aa25ff2c49642b194d7196b7cc"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.671839 2779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:50.672751 2795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.672959 2779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:50.673072 2779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "78ca68890fca41d084ac33d482b77a5b"
format_stamp: "Formatted at 2026-08-12 00:13:50 on dist-test-slave-zpfg"
server_key: "253e9e5a863d254d1cb9662cd292ea24"
server_key_iv: "7da2c6aa25ff2c49642b194d7196b7cc"
server_key_version: "encryptionkey@0"
I20260812 00:13:50.673152 2779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:50.689603 2779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:50.692478 2779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:50.692680 2779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:50.692994 2779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:50.693348 2779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:50.693406 2779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.693449 2779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:50.693473 2779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:50.701699 2779 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:38641
I20260812 00:13:50.701751 2913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:38641 every 8 connection(s)
I20260812 00:13:50.702322 2779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:13:50.705713 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 2779
I20260812 00:13:50.705855 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:13:50.706029 16783 external_mini_cluster.cc:1428] Setting key 0f14b470ac170f6736934c06f8b8c00e
Aug 12 00:13:50 dist-test-slave-zpfg krb5kdc[2372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493630, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:50.713814 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.703360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:47289 (local address 127.16.99.254:40475)
0812 00:13:50.703421 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.703423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.703715 (+ 292us) server_negotiation.cc:407] Connection header received
0812 00:13:50.704557 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.704559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.704574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.704716 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:50.706069 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.706956 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.708026 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.708243 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.708327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.710669 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:50.710670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:50.710670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:50.710678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:50.712079 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.712748 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.712749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.712750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.712816 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.713102 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.713111 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.713112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.713241 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:50.713314 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.713614 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.713708 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 00:13:50.714064 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.703353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40475 (local address 127.16.99.195:47289)
0812 00:13:50.703514 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.703556 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.704422 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.704821 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.704825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.705686 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:50.705953 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.705959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.707055 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.707056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.707930 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.707933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.708335 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.708336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.708469 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.708996 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:50.709005 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:50.710561 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:50.712354 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:50.712364 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:50.712366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:50.712660 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:50.712902 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:50.712903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:50.712903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:50.713024 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:50.713316 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:50.713325 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:50.713472 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.713590 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.713784 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260812 00:13:50.714622 2914 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40475
I20260812 00:13:50.714758 2914 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:50.715003 2914 heartbeater.cc:511] Master 127.16.99.254:40475 requested a full tablet report, sending...
I20260812 00:13:50.715602 2416 ts_manager.cc:194] Registered new tserver with Master: 78ca68890fca41d084ac33d482b77a5b (127.16.99.195:38641)
I20260812 00:13:50.716040 2416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:47289
I20260812 00:13:50.718498 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:50.726785 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.720106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.16.99.254:40475)
0812 00:13:50.720319 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.720321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.720335 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:13:50.720412 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.720413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.720423 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.720478 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:50.721071 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.721667 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.722427 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.722633 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.722696 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.723968 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:50.723969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:50.723970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:50.723976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:50.725593 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.725958 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.725959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.725959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.726013 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:50.726201 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:50.726201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:50.726202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:50.726420 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:50.726500 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.726587 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.726665 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:13:50.728722 2416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:50.730472 2416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:50.737834 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.733311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:38641 (local address 127.0.0.1:41236)
0812 00:13:50.733722 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.733752 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.733860 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.734301 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.734302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.734304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:50.734378 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.734380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.735378 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.735380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.736598 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.736600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.737474 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.737475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.737570 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.737587 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.737661 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.737722 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":896,"thread_start_us":111,"threads_started":1}
I20260812 00:13:50.737972 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.733308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:40875 (local address 127.0.0.1:52420)
0812 00:13:50.733702 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.733751 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.733831 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.734185 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.734189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.734191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:50.734268 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.734270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.735702 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.735703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.736772 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.736775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.737684 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.737685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.737752 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.737756 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.737816 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.737859 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":59,"threads_started":1}
I20260812 00:13:50.738240 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.733415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.16.99.195:38641)
0812 00:13:50.733614 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.733620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.733827 (+ 207us) server_negotiation.cc:407] Connection header received
0812 00:13:50.733986 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.733989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.734006 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.734214 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:50.734446 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.735259 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.736687 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.737385 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.737463 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.737532 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.738084 (+ 552us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.738133 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":110,"threads_started":1}
I20260812 00:13:50.738441 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.733608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46395 (local address 127.0.0.1:49780)
0812 00:13:50.734105 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.734116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.734168 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.734490 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.734491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.734493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:50.734556 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.734558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.735866 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.735867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.737132 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.737135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.738181 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.738183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.738260 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.738264 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.738311 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.738349 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":71,"threads_started":1}
I20260812 00:13:50.738438 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.733436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.16.99.194:40875)
0812 00:13:50.733667 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.733670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.733942 (+ 272us) server_negotiation.cc:407] Connection header received
0812 00:13:50.733986 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.733989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.734006 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.734082 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:50.734380 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.735576 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.736878 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.737584 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.737661 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.737730 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.737827 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.737878 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":146,"threads_started":1}
I20260812 00:13:50.738782 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.733854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.16.99.193:46395)
0812 00:13:50.734046 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.734051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.734226 (+ 175us) server_negotiation.cc:407] Connection header received
0812 00:13:50.734274 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.734276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.734293 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.734398 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:50.734648 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.735751 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.737231 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.738016 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.738089 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.738148 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.738650 (+ 502us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.738691 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260812 00:13:50.739071 2846 tablet_service.cc:1467] Processing CreateTablet for tablet 81acbeee52e2478587107371edde2ef7 (DEFAULT_TABLE table=test-table [id=6004138432c64e52aede04883bc8aa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:50.739504 2846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81acbeee52e2478587107371edde2ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:50.739401 2703 tablet_service.cc:1467] Processing CreateTablet for tablet 81acbeee52e2478587107371edde2ef7 (DEFAULT_TABLE table=test-table [id=6004138432c64e52aede04883bc8aa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:50.739430 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 81acbeee52e2478587107371edde2ef7 (DEFAULT_TABLE table=test-table [id=6004138432c64e52aede04883bc8aa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:50.739744 2703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81acbeee52e2478587107371edde2ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:50.739804 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81acbeee52e2478587107371edde2ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:50.742795 2934 tablet_bootstrap.cc:492] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6: Bootstrap starting.
I20260812 00:13:50.743055 2936 tablet_bootstrap.cc:492] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Bootstrap starting.
I20260812 00:13:50.743732 2934 tablet_bootstrap.cc:654] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:50.743732 2936 tablet_bootstrap.cc:654] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:50.744421 2936 log.cc:824] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:50.744427 2934 log.cc:824] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:50.744798 2935 tablet_bootstrap.cc:492] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Bootstrap starting.
I20260812 00:13:50.745281 2936 tablet_bootstrap.cc:492] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: No bootstrap required, opened a new log
I20260812 00:13:50.745373 2936 ts_tablet_manager.cc:1397] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:13:50.745476 2935 tablet_bootstrap.cc:654] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:50.745507 2934 tablet_bootstrap.cc:492] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6: No bootstrap required, opened a new log
I20260812 00:13:50.745662 2934 ts_tablet_manager.cc:1397] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:13:50.746138 2935 log.cc:824] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:50.747217 2936 raft_consensus.cc:348] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.747391 2936 raft_consensus.cc:374] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:50.747440 2936 raft_consensus.cc:729] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ca68890fca41d084ac33d482b77a5b, State: Initialized, Role: FOLLOWER
I20260812 00:13:50.747371 2934 raft_consensus.cc:348] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.747583 2934 raft_consensus.cc:374] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:50.747563 2936 consensus_queue.cc:260] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.747633 2934 raft_consensus.cc:729] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f6b9559b84461894a88646c3761be6, State: Initialized, Role: FOLLOWER
I20260812 00:13:50.747749 2934 consensus_queue.cc:260] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.747910 2936 ts_tablet_manager.cc:1428] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:13:50.747921 2914 heartbeater.cc:503] Master 127.16.99.254:40475 was elected leader, sending a full tablet report...
I20260812 00:13:50.748075 2934 ts_tablet_manager.cc:1428] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:13:50.748123 2631 heartbeater.cc:503] Master 127.16.99.254:40475 was elected leader, sending a full tablet report...
I20260812 00:13:50.749091 2935 tablet_bootstrap.cc:492] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: No bootstrap required, opened a new log
I20260812 00:13:50.749171 2935 ts_tablet_manager.cc:1397] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260812 00:13:50.750918 2935 raft_consensus.cc:348] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.751070 2935 raft_consensus.cc:374] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:50.751094 2935 raft_consensus.cc:729] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416fb4c81de84f4191c4048609b26541, State: Initialized, Role: FOLLOWER
I20260812 00:13:50.751257 2935 consensus_queue.cc:260] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.751533 2935 ts_tablet_manager.cc:1428] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:13:50.751602 2773 heartbeater.cc:503] Master 127.16.99.254:40475 was elected leader, sending a full tablet report...
W20260812 00:13:50.766530 2774 tablet.cc:2369] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:13:50.829541 2632 tablet.cc:2369] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:50.950218 2942 raft_consensus.cc:482] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:50.950415 2942 raft_consensus.cc:504] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.950865 2942 leader_election.cc:288] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78ca68890fca41d084ac33d482b77a5b (127.16.99.195:38641), 07f6b9559b84461894a88646c3761be6 (127.16.99.193:46395)
W20260812 00:13:50.953111 2915 tablet.cc:2369] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:50.954805 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.951191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:54653 (local address 127.16.99.195:38641)
0812 00:13:50.951292 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.951295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.951368 (+ 73us) server_negotiation.cc:407] Connection header received
0812 00:13:50.951419 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.951420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.951430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.951484 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:50.951818 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.952533 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.953780 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.954329 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.954425 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.954436 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.954667 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.954715 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 00:13:50.954978 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.951055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46395 (local address 127.16.99.194:57149)
0812 00:13:50.951110 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.951125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.951183 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.951490 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.951491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.951493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:50.951575 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.951577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.952567 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.952569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.953922 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.953925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.954709 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.954710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.954786 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.954789 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.954838 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.954887 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260812 00:13:50.955226 2943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.951062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:38641 (local address 127.16.99.194:54653)
0812 00:13:50.951285 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.951296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:50.951358 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:50.951662 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:50.951663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:50.951664 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:50.951725 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.951727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.952601 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.952602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.953681 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:50.953684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.954470 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.954471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:50.954551 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.954557 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:13:50.955081 (+ 524us) client_negotiation.cc:240] Negotiation successful
0812 00:13:50.955131 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":40,"thread_start_us":108,"threads_started":1}
I20260812 00:13:50.955425 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.951251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:57149 (local address 127.16.99.193:46395)
0812 00:13:50.951330 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.951332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.951361 (+ 29us) server_negotiation.cc:407] Connection header received
0812 00:13:50.951394 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.951395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.951424 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.951484 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:50.951689 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.952450 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.954022 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.954599 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.954672 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.954680 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.955308 (+ 628us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.955365 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:13:50.955762 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acbeee52e2478587107371edde2ef7" candidate_uuid: "416fb4c81de84f4191c4048609b26541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f6b9559b84461894a88646c3761be6" is_pre_election: true
I20260812 00:13:50.955932 2578 raft_consensus.cc:2454] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416fb4c81de84f4191c4048609b26541 in term 0.
I20260812 00:13:50.956276 2654 leader_election.cc:302] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f6b9559b84461894a88646c3761be6, 416fb4c81de84f4191c4048609b26541; no voters:
I20260812 00:13:50.956432 2942 raft_consensus.cc:2781] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:50.956512 2942 raft_consensus.cc:482] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:50.956562 2942 raft_consensus.cc:3037] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:50.956904 2866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acbeee52e2478587107371edde2ef7" candidate_uuid: "416fb4c81de84f4191c4048609b26541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca68890fca41d084ac33d482b77a5b" is_pre_election: true
I20260812 00:13:50.957074 2866 raft_consensus.cc:2454] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416fb4c81de84f4191c4048609b26541 in term 0.
I20260812 00:13:50.957697 2942 raft_consensus.cc:504] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.957945 2942 leader_election.cc:288] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [CANDIDATE]: Term 1 election: Requested vote from peers 78ca68890fca41d084ac33d482b77a5b (127.16.99.195:38641), 07f6b9559b84461894a88646c3761be6 (127.16.99.193:46395)
I20260812 00:13:50.958026 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acbeee52e2478587107371edde2ef7" candidate_uuid: "416fb4c81de84f4191c4048609b26541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f6b9559b84461894a88646c3761be6"
I20260812 00:13:50.958101 2866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81acbeee52e2478587107371edde2ef7" candidate_uuid: "416fb4c81de84f4191c4048609b26541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca68890fca41d084ac33d482b77a5b"
I20260812 00:13:50.958110 2578 raft_consensus.cc:3037] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:50.958171 2866 raft_consensus.cc:3037] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:50.959108 2866 raft_consensus.cc:2454] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416fb4c81de84f4191c4048609b26541 in term 1.
I20260812 00:13:50.959108 2578 raft_consensus.cc:2454] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416fb4c81de84f4191c4048609b26541 in term 1.
I20260812 00:13:50.959303 2654 leader_election.cc:302] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f6b9559b84461894a88646c3761be6, 416fb4c81de84f4191c4048609b26541; no voters:
I20260812 00:13:50.959532 2942 raft_consensus.cc:2781] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:50.959779 2942 raft_consensus.cc:686] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 1 LEADER]: Becoming Leader. State: Replica: 416fb4c81de84f4191c4048609b26541, State: Running, Role: LEADER
I20260812 00:13:50.959898 2942 consensus_queue.cc:237] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } }
I20260812 00:13:50.963023 2415 catalog_manager.cc:5495] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 reported cstate change: term changed from 0 to 1, leader changed from <none> to 416fb4c81de84f4191c4048609b26541 (127.16.99.194). New cstate: current_term: 1 leader_uuid: "416fb4c81de84f4191c4048609b26541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416fb4c81de84f4191c4048609b26541" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 40875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:50.971539 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.969038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.16.99.194:40875)
0812 00:13:50.969087 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:50.969089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:50.969097 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:13:50.969258 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:50.969259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:50.969278 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:50.969344 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:50.969460 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.969918 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.970808 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:50.970994 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:50.971053 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:50.971188 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:50.971293 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:50.971358 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:50.971393 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 00:13:50.971411 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:13:50.973603 2578 raft_consensus.cc:1261] T 81acbeee52e2478587107371edde2ef7 P 07f6b9559b84461894a88646c3761be6 [term 1 FOLLOWER]: Refusing update from remote peer 416fb4c81de84f4191c4048609b26541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:50.973645 2866 raft_consensus.cc:1261] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 1 FOLLOWER]: Refusing update from remote peer 416fb4c81de84f4191c4048609b26541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:50.973996 2944 consensus_queue.cc:1035] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ca68890fca41d084ac33d482b77a5b" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 38641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:50.974143 2942 consensus_queue.cc:1035] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f6b9559b84461894a88646c3761be6" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:50.977872 2949 mvcc.cc:205] Tried to move back new op lower bound from 7317477912464367616 to 7317477912412340224. Current Snapshot: MvccSnapshot[applied={T|T < 7317477912464367616 or (T in {7317477912464367616})}]
I20260812 00:13:50.979274 2416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:50.979394 2416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:50.980427 2416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b69b936bc4845ac9ab183b06a537ab4: Sending DeleteTablet for 3 replicas of tablet 81acbeee52e2478587107371edde2ef7
I20260812 00:13:50.980788 2846 tablet_service.cc:1514] Processing DeleteTablet for tablet 81acbeee52e2478587107371edde2ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:50 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:41236
I20260812 00:13:50.980849 2703 tablet_service.cc:1514] Processing DeleteTablet for tablet 81acbeee52e2478587107371edde2ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:50 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:52420
I20260812 00:13:50.980924 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 81acbeee52e2478587107371edde2ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:50 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:49780
I20260812 00:13:50.981242 2957 tablet_replica.cc:330] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: stopping tablet replica
I20260812 00:13:50.981307 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 2489
I20260812 00:13:50.981410 2957 raft_consensus.cc:2229] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:50.981525 2957 raft_consensus.cc:2258] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:50.982993 2957 ts_tablet_manager.cc:1905] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:50.984898 2957 ts_tablet_manager.cc:1918] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:50.984992 2957 log.cc:1196] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/81acbeee52e2478587107371edde2ef7
I20260812 00:13:50.985302 2957 ts_tablet_manager.cc:1939] T 81acbeee52e2478587107371edde2ef7 P 78ca68890fca41d084ac33d482b77a5b: Deleting consensus metadata
I20260812 00:13:50.985813 2404 catalog_manager.cc:4841] TS 78ca68890fca41d084ac33d482b77a5b (127.16.99.195:38641): tablet 81acbeee52e2478587107371edde2ef7 (table test-table [id=6004138432c64e52aede04883bc8aa0a]) successfully deleted
I20260812 00:13:50.986032 2958 tablet_replica.cc:330] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: stopping tablet replica
I20260812 00:13:50.986156 2958 raft_consensus.cc:2229] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:13:50.986260 2958 raft_consensus.cc:2258] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:50.986728 2958 ts_tablet_manager.cc:1905] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:50.988489 2958 ts_tablet_manager.cc:1918] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:50.988575 2958 log.cc:1196] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/81acbeee52e2478587107371edde2ef7
I20260812 00:13:50.988857 2958 ts_tablet_manager.cc:1939] T 81acbeee52e2478587107371edde2ef7 P 416fb4c81de84f4191c4048609b26541: Deleting consensus metadata
I20260812 00:13:50.989449 2403 catalog_manager.cc:4841] TS 416fb4c81de84f4191c4048609b26541 (127.16.99.194:40875): tablet 81acbeee52e2478587107371edde2ef7 (table test-table [id=6004138432c64e52aede04883bc8aa0a]) successfully deleted
W20260812 00:13:50.998656 2403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:46395 (error 108)
I20260812 00:13:50.999033 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 2637
I20260812 00:13:50.999339 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:50.999205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46395 (local address 127.0.0.1:49784)
0812 00:13:50.999264 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:50.999303 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 00:13:51.000078 2403 catalog_manager.cc:4584] TS 07f6b9559b84461894a88646c3761be6 (127.16.99.193:46395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81acbeee52e2478587107371edde2ef7: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46395: connect: Connection refused (error 111)
I20260812 00:13:51.009277 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 2779
I20260812 00:13:51.016105 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 2385
2026-08-12T00:13:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2386 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T00:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:51Z Disabled control of system clock
I20260812 00:13:51.048990 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37279
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:37279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:51.141490 2964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:51.141738 2964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:51.141760 2964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:51.143178 2964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:51.143232 2964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:51.143245 2964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:51.143260 2964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:51.143272 2964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:51.144790 2964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:37279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:51.145081 2964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:51.145416 2964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:51.151243 2969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:51.151223 2970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:51.151351 2972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:51.151551 2964 server_base.cc:1034] running on GCE node
I20260812 00:13:51.151811 2964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:51.152153 2964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:51.153337 2964 hybrid_clock.cc:648] HybridClock initialized: now 1786493631153317 us; error 51 us; skew 500 ppm
I20260812 00:13:51.154959 2964 webserver.cc:460] Webserver started at http://127.16.99.254:43589/ using document root <none> and password file <none>
I20260812 00:13:51.155263 2964 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:51.155340 2964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:51.155509 2964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:51.156654 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "888b80765fa74788924f8b4c06b401a7"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "fd9d17d378cdbdee759eb3c768dfd12b"
server_key_iv: "22e1ad7f7c81a709e16db380acbc8ee0"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.157047 2964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "888b80765fa74788924f8b4c06b401a7"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "fd9d17d378cdbdee759eb3c768dfd12b"
server_key_iv: "22e1ad7f7c81a709e16db380acbc8ee0"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.158588 2964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:13:51.159754 2978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.160053 2964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:51.160192 2964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "888b80765fa74788924f8b4c06b401a7"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "fd9d17d378cdbdee759eb3c768dfd12b"
server_key_iv: "22e1ad7f7c81a709e16db380acbc8ee0"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.160286 2964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:51.175809 2964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:51.176218 2964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:51.176406 2964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:51.180612 2964 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:37279
I20260812 00:13:51.180676 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:37279 every 8 connection(s)
I20260812 00:13:51.181078 2964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:51.182245 3037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:51.184526 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 2964
I20260812 00:13:51.184629 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:13:51.184860 16783 external_mini_cluster.cc:1428] Setting key d7b73df952e797c45fb499ed42f5fb01
I20260812 00:13:51.184846 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7: Bootstrap starting.
I20260812 00:13:51.185665 3037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:51.185992 3037 log.cc:824] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:51.186925 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7: No bootstrap required, opened a new log
I20260812 00:13:51.189213 3037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888b80765fa74788924f8b4c06b401a7" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37279 } }
I20260812 00:13:51.189416 3037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:51.189452 3037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888b80765fa74788924f8b4c06b401a7, State: Initialized, Role: FOLLOWER
I20260812 00:13:51.189579 3037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888b80765fa74788924f8b4c06b401a7" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37279 } }
I20260812 00:13:51.189656 3037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:51.189702 3037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:51.189864 3037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:51.191057 3037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888b80765fa74788924f8b4c06b401a7" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37279 } }
I20260812 00:13:51.191218 3037 leader_election.cc:302] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888b80765fa74788924f8b4c06b401a7; no voters:
I20260812 00:13:51.191429 3037 leader_election.cc:288] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:51.191538 3042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:51.191787 3037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:51.191820 3042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [term 1 LEADER]: Becoming Leader. State: Replica: 888b80765fa74788924f8b4c06b401a7, State: Running, Role: LEADER
I20260812 00:13:51.192009 3042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888b80765fa74788924f8b4c06b401a7" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37279 } }
I20260812 00:13:51.192173 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.185460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.16.99.254:37279)
0812 00:13:51.185668 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.185676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.185702 (+ 26us) server_negotiation.cc:407] Connection header received
0812 00:13:51.186259 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.186267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.186328 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.186499 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:51.186755 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.188164 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.189052 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.189674 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.190505 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.190968 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:51.190972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:51.190973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:51.191220 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:51.191272 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:51.191290 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:51.191631 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.191670 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.191770 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260812 00:13:51.192366 3042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888b80765fa74788924f8b4c06b401a7. Latest consensus state: current_term: 1 leader_uuid: "888b80765fa74788924f8b4c06b401a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888b80765fa74788924f8b4c06b401a7" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37279 } } }
I20260812 00:13:51.192445 3042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:51.192430 3043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "888b80765fa74788924f8b4c06b401a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888b80765fa74788924f8b4c06b401a7" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37279 } } }
I20260812 00:13:51.193082 3043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:51.193094 3047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:51.193773 3047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:51.198585 3047 catalog_manager.cc:1347] Generated new cluster ID: 8cfbe92fcfcb4278b091f8b4f0e3faaf
I20260812 00:13:51.198673 3047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:51.223877 3047 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:51.224660 3047 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:51.233461 3047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7: Generated new TSK 0
I20260812 00:13:51.233696 3047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:13:51.241796 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37279
--builtin_ntp_servers=127.16.99.212:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:51.330636 3063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:51.330883 3063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:51.330909 3063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:51.332248 3063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:51.332306 3063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:51.332334 3063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:51.333845 3063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:51.334131 3063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:51.334551 3063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:51.340149 3068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:51.340086 3069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:51.340086 3071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:51.340924 3063 server_base.cc:1034] running on GCE node
I20260812 00:13:51.341190 3063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:51.341465 3063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:51.342620 3063 hybrid_clock.cc:648] HybridClock initialized: now 1786493631342593 us; error 42 us; skew 500 ppm
I20260812 00:13:51.344086 3063 webserver.cc:460] Webserver started at http://127.16.99.193:42279/ using document root <none> and password file <none>
I20260812 00:13:51.344317 3063 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:51.344362 3063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:51.344457 3063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:51.345589 3063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "840b26cc3cb547cc9ce30a4e255c275b"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "1bc02c2cf61b2635fba31395fa1721f5"
server_key_iv: "3d23fb2305066e6ac964b1603c751585"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.345921 3063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "840b26cc3cb547cc9ce30a4e255c275b"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "1bc02c2cf61b2635fba31395fa1721f5"
server_key_iv: "3d23fb2305066e6ac964b1603c751585"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.347546 3063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:13:51.348488 3077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.348708 3063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:51.348857 3063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "840b26cc3cb547cc9ce30a4e255c275b"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "1bc02c2cf61b2635fba31395fa1721f5"
server_key_iv: "3d23fb2305066e6ac964b1603c751585"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.348950 3063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:51.361480 3063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:51.361851 3063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:51.362017 3063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:51.362282 3063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:51.362663 3063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:51.362787 3063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.362844 3063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:51.362875 3063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.370067 3063 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:45005
I20260812 00:13:51.370234 3198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:45005 every 8 connection(s)
I20260812 00:13:51.370597 3063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:51.376521 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.371468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:55981 (local address 127.16.99.254:37279)
0812 00:13:51.371538 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.371540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.371815 (+ 275us) server_negotiation.cc:407] Connection header received
0812 00:13:51.372374 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.372376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.372388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.372452 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:51.373182 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.373703 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.374848 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.375066 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.375146 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.376179 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:51.376180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:51.376181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:51.376187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:51.376205 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:51.376210 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:51.376291 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.376400 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.376432 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:13:51.376803 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.371455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37279 (local address 127.16.99.193:55981)
0812 00:13:51.371648 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.371684 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.372259 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.372531 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.372535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.372550 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:51.373079 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.373084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.373805 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.373806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.374752 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.374755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.375155 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.375156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.375315 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.376007 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:51.376007 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:51.376039 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:51.376075 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:51.376301 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:51.376309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:51.376312 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.376402 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.376514 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260812 00:13:51.377398 3199 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37279
I20260812 00:13:51.377524 3199 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:51.377722 3199 heartbeater.cc:511] Master 127.16.99.254:37279 requested a full tablet report, sending...
I20260812 00:13:51.378158 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 3063
I20260812 00:13:51.378295 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:13:51.378528 2996 ts_manager.cc:194] Registered new tserver with Master: 840b26cc3cb547cc9ce30a4e255c275b (127.16.99.193:45005)
I20260812 00:13:51.378561 16783 external_mini_cluster.cc:1428] Setting key 31ea0606dc310c1fd18939bfd03d0bdf
I20260812 00:13:51.379292 2996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.193:55981
I20260812 00:13:51.379958 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37279
--builtin_ntp_servers=127.16.99.212:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:51.471415 3202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:51.471697 3202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:51.471726 3202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:51.473037 3202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:51.473105 3202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:51.473141 3202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:51.474723 3202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:51.475039 3202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:51.475423 3202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:51.481719 3208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:51.481757 3210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:51.481760 3207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:51.481992 3202 server_base.cc:1034] running on GCE node
I20260812 00:13:51.482209 3202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:51.482574 3202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:51.483783 3202 hybrid_clock.cc:648] HybridClock initialized: now 1786493631483758 us; error 44 us; skew 500 ppm
I20260812 00:13:51.485392 3202 webserver.cc:460] Webserver started at http://127.16.99.194:36031/ using document root <none> and password file <none>
I20260812 00:13:51.485724 3202 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:51.485809 3202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:51.485991 3202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:51.487293 3202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "12f4ca4fc8919abca4edceac67ed71f4"
server_key_iv: "8bb77a724ac06c535a1bd8a07376b1ff"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.487682 3202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "12f4ca4fc8919abca4edceac67ed71f4"
server_key_iv: "8bb77a724ac06c535a1bd8a07376b1ff"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.489159 3202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 00:13:51.490134 3217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.490473 3202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:51.490576 3202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "12f4ca4fc8919abca4edceac67ed71f4"
server_key_iv: "8bb77a724ac06c535a1bd8a07376b1ff"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.490638 3202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:51.502735 3202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:51.503074 3202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:51.503190 3202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:51.503407 3202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:51.503796 3202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:51.503844 3202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.503867 3202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:51.503880 3202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.510900 3202 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:41225
I20260812 00:13:51.510949 3336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:41225 every 8 connection(s)
I20260812 00:13:51.511368 3202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:13:51.516693 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 3202
I20260812 00:13:51.516824 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:13:51.517067 16783 external_mini_cluster.cc:1428] Setting key 38dee065e2bbb0968ec7e4864dc75bde
I20260812 00:13:51.517581 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.512372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:42983 (local address 127.16.99.254:37279)
0812 00:13:51.512445 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.512447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.512682 (+ 235us) server_negotiation.cc:407] Connection header received
0812 00:13:51.513258 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.513259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.513271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.513333 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:51.514111 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.514767 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.515843 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.516035 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.516114 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.517234 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:51.517236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:51.517236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:51.517242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:51.517260 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:51.517265 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:51.517348 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.517466 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.517500 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:13:51.517822 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.512340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37279 (local address 127.16.99.194:42983)
0812 00:13:51.512549 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.512586 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.513131 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.513425 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.513428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.513437 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:51.514017 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.514022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.514878 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.514879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.515744 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.515747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.516170 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.516172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.516323 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.517073 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:51.517073 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:51.517104 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:51.517131 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:51.517349 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:51.517358 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:51.517362 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.517438 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.517559 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260812 00:13:51.518318 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37279
--builtin_ntp_servers=127.16.99.212:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 00:13:51.518438 3337 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37279
I20260812 00:13:51.518543 3337 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:51.518739 3337 heartbeater.cc:511] Master 127.16.99.254:37279 requested a full tablet report, sending...
I20260812 00:13:51.519134 2996 ts_manager.cc:194] Registered new tserver with Master: 4ac8fb7cfdc94f58b31f6e59ca5b7449 (127.16.99.194:41225)
I20260812 00:13:51.519528 2996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.194:42983
W20260812 00:13:51.607076 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:51.607321 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:51.607347 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:51.608589 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:51.608633 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:51.608659 3341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:51.610090 3341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:51.610322 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:51.610716 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:51.616248 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:51.616240 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:51.616330 3341 server_base.cc:1034] running on GCE node
W20260812 00:13:51.616241 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:51.616742 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:51.617036 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:51.618254 3341 hybrid_clock.cc:648] HybridClock initialized: now 1786493631618235 us; error 34 us; skew 500 ppm
I20260812 00:13:51.619951 3341 webserver.cc:460] Webserver started at http://127.16.99.195:42025/ using document root <none> and password file <none>
I20260812 00:13:51.620272 3341 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:51.620325 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:51.620507 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:51.621778 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "e6beb214562c4740beea5f35c4689f86"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "d008c88df22837451280a9b7fc4ca8d5"
server_key_iv: "323d30920708c5238db6ae15fb98ec34"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.622169 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "e6beb214562c4740beea5f35c4689f86"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "d008c88df22837451280a9b7fc4ca8d5"
server_key_iv: "323d30920708c5238db6ae15fb98ec34"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.623708 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:51.624632 3355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.624854 3341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:51.624961 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "e6beb214562c4740beea5f35c4689f86"
format_stamp: "Formatted at 2026-08-12 00:13:51 on dist-test-slave-zpfg"
server_key: "d008c88df22837451280a9b7fc4ca8d5"
server_key_iv: "323d30920708c5238db6ae15fb98ec34"
server_key_version: "encryptionkey@0"
I20260812 00:13:51.625047 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:51.642812 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:51.643169 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:51.643332 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:51.643590 3341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:51.643972 3341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:51.644030 3341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.644065 3341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:51.644096 3341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:51.650436 3341 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:42619
I20260812 00:13:51.650492 3477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:42619 every 8 connection(s)
I20260812 00:13:51.650923 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:13:51.654922 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 3341
I20260812 00:13:51.655021 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:13:51.655297 16783 external_mini_cluster.cc:1428] Setting key fa22e2a7d8021d6f38aa839dd66682ff
I20260812 00:13:51.656855 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.651749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:37927 (local address 127.16.99.254:37279)
0812 00:13:51.651826 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.651828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.652067 (+ 239us) server_negotiation.cc:407] Connection header received
0812 00:13:51.652646 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.652647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.652660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.652724 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:51.653460 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.653955 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.655236 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.655480 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.655556 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.656594 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:51.656595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:51.656596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:51.656601 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:51.656617 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:51.656621 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:51.656669 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.656748 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.656778 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 00:13:51.657104 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.651741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37279 (local address 127.16.99.195:37927)
0812 00:13:51.651937 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.651971 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.652553 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.652836 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.652839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.652848 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:51.653371 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.653375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.654043 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.654043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.655134 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.655138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.655570 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.655571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.655788 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.656431 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:51.656432 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:51.656473 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:51.656503 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:51.656669 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:51.656675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:51.656678 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.656753 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.656863 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260812 00:13:51.657620 3478 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37279
I20260812 00:13:51.657729 3478 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:51.657925 3478 heartbeater.cc:511] Master 127.16.99.254:37279 requested a full tablet report, sending...
I20260812 00:13:51.658393 2996 ts_manager.cc:194] Registered new tserver with Master: e6beb214562c4740beea5f35c4689f86 (127.16.99.195:42619)
I20260812 00:13:51.658859 2996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.195:37927
I20260812 00:13:51.667868 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:51.672642 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.669621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.16.99.254:37279)
0812 00:13:51.669692 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.669694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.669800 (+ 106us) server_negotiation.cc:407] Connection header received
0812 00:13:51.669846 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.669847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.669857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.669910 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:51.670065 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.670671 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.671497 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.671681 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.671743 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.672370 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:51.672371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:51.672371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:51.672376 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:51.672389 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:51.672393 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:51.672441 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.672505 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.672525 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 00:13:51.674582 2996 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:51.676301 2996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:51.683465 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.679149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.16.99.193:45005)
0812 00:13:51.679781 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.679786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.679796 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:13:51.679844 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.679846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.679862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.679933 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:51.680136 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.681273 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.682439 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.683023 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.683117 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.683235 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.683312 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.683353 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":87,"threads_started":1}
I20260812 00:13:51.683440 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.678980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45005 (local address 127.0.0.1:52180)
0812 00:13:51.679318 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.679360 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.679436 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.680005 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.680006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.680008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:51.680067 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.680069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.681369 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.681374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.682341 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.682344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.683117 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.683118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.683202 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.683215 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.683295 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.683354 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":69,"threads_started":1}
I20260812 00:13:51.683599 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.679194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:42619 (local address 127.0.0.1:48986)
0812 00:13:51.679540 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.679548 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.679627 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.679912 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.679913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.679915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:51.679969 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.679970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.680904 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.680905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.682175 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.682178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.683337 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.683338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.683408 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.683412 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.683459 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.683509 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":93,"threads_started":1}
I20260812 00:13:51.683701 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.678934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:41225 (local address 127.0.0.1:43254)
0812 00:13:51.679312 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.679360 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.679436 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.679817 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.679820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.679822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:51.679904 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.679907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.681304 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.681305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.682487 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.682490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.683413 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.683413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.683475 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.683481 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.683519 (+ 38us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.683552 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":86,"thread_start_us":92,"threads_started":1}
I20260812 00:13:51.683949 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.679063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.16.99.194:41225)
0812 00:13:51.679607 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.679611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.679622 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:51.679669 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.679671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.679687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.679993 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:51.680069 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.681200 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.682574 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.683320 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.683649 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.683772 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.683797 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.683839 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":128,"threads_started":1}
I20260812 00:13:51.684167 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.679296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.16.99.195:42619)
0812 00:13:51.679474 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.679480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.679634 (+ 154us) server_negotiation.cc:407] Connection header received
0812 00:13:51.679783 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.679786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.679826 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.679919 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:51.680051 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.680810 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.682284 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.683110 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.683187 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.683294 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.683703 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.683751 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260812 00:13:51.684765 3127 tablet_service.cc:1467] Processing CreateTablet for tablet ee08c2bdac31491cb781ce93f56425d4 (DEFAULT_TABLE table=test-table [id=9c5bd0183ab846c68719844cf1c7b3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:51.685209 3127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08c2bdac31491cb781ce93f56425d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:51.687103 3267 tablet_service.cc:1467] Processing CreateTablet for tablet ee08c2bdac31491cb781ce93f56425d4 (DEFAULT_TABLE table=test-table [id=9c5bd0183ab846c68719844cf1c7b3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:51.687374 3407 tablet_service.cc:1467] Processing CreateTablet for tablet ee08c2bdac31491cb781ce93f56425d4 (DEFAULT_TABLE table=test-table [id=9c5bd0183ab846c68719844cf1c7b3bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:51.687546 3267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08c2bdac31491cb781ce93f56425d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:51.687727 3407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee08c2bdac31491cb781ce93f56425d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:51.688706 3493 tablet_bootstrap.cc:492] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b: Bootstrap starting.
I20260812 00:13:51.689467 3493 tablet_bootstrap.cc:654] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:51.689810 3493 log.cc:824] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:51.690768 3493 tablet_bootstrap.cc:492] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b: No bootstrap required, opened a new log
I20260812 00:13:51.690871 3493 ts_tablet_manager.cc:1397] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:13:51.691232 3496 tablet_bootstrap.cc:492] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Bootstrap starting.
I20260812 00:13:51.692006 3496 tablet_bootstrap.cc:654] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:51.692279 3496 log.cc:824] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:51.692632 3495 tablet_bootstrap.cc:492] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Bootstrap starting.
I20260812 00:13:51.693320 3496 tablet_bootstrap.cc:492] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: No bootstrap required, opened a new log
I20260812 00:13:51.693404 3496 ts_tablet_manager.cc:1397] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:13:51.693430 3495 tablet_bootstrap.cc:654] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:51.693831 3495 log.cc:824] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:51.694490 3493 raft_consensus.cc:348] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.694713 3493 raft_consensus.cc:374] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:51.694769 3493 raft_consensus.cc:729] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840b26cc3cb547cc9ce30a4e255c275b, State: Initialized, Role: FOLLOWER
I20260812 00:13:51.694892 3495 tablet_bootstrap.cc:492] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: No bootstrap required, opened a new log
I20260812 00:13:51.694962 3495 ts_tablet_manager.cc:1397] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:13:51.694928 3493 consensus_queue.cc:260] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.695235 3493 ts_tablet_manager.cc:1428] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260812 00:13:51.695359 3199 heartbeater.cc:503] Master 127.16.99.254:37279 was elected leader, sending a full tablet report...
I20260812 00:13:51.696051 3496 raft_consensus.cc:348] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.696250 3496 raft_consensus.cc:374] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:51.696291 3496 raft_consensus.cc:729] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac8fb7cfdc94f58b31f6e59ca5b7449, State: Initialized, Role: FOLLOWER
I20260812 00:13:51.696409 3496 consensus_queue.cc:260] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.696702 3496 ts_tablet_manager.cc:1428] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:13:51.696833 3337 heartbeater.cc:503] Master 127.16.99.254:37279 was elected leader, sending a full tablet report...
I20260812 00:13:51.697824 3495 raft_consensus.cc:348] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.697953 3495 raft_consensus.cc:374] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:51.697974 3495 raft_consensus.cc:729] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6beb214562c4740beea5f35c4689f86, State: Initialized, Role: FOLLOWER
I20260812 00:13:51.698093 3495 consensus_queue.cc:260] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.698391 3495 ts_tablet_manager.cc:1428] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:13:51.698452 3478 heartbeater.cc:503] Master 127.16.99.254:37279 was elected leader, sending a full tablet report...
W20260812 00:13:51.762125 3338 tablet.cc:2369] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:51.806427 3501 raft_consensus.cc:482] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:51.806713 3501 raft_consensus.cc:504] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.807159 3501 leader_election.cc:288] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ac8fb7cfdc94f58b31f6e59ca5b7449 (127.16.99.194:41225), 840b26cc3cb547cc9ce30a4e255c275b (127.16.99.193:45005)
I20260812 00:13:51.810683 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.807421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45005 (local address 127.16.99.195:52053)
0812 00:13:51.807516 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.807530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.807631 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.807832 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.807832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.807835 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:51.807915 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.807917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.808739 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.808740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.809680 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.809683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.810385 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.810386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.810462 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.810465 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.810546 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.810593 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 00:13:51.810683 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.807559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40907 (local address 127.16.99.194:41225)
0812 00:13:51.807631 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.807634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.807921 (+ 287us) server_negotiation.cc:407] Connection header received
0812 00:13:51.807955 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.807956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.807969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.808020 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:51.808189 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.808673 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.809836 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.810220 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.810286 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.810320 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.810522 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.810594 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:13:51.810704 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.807563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:52053 (local address 127.16.99.193:45005)
0812 00:13:51.807686 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.807688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.807699 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:51.807734 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.807735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.807747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.807818 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:51.808040 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.808644 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.809793 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.810230 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.810332 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.810367 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.810574 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.810634 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:13:51.810738 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.807438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:41225 (local address 127.16.99.195:40907)
0812 00:13:51.807841 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.807851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:51.807912 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:51.808037 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:51.808037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:51.808039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:51.808119 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.808121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.808792 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.808793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.809694 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:51.809697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.810323 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.810324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:51.810437 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.810441 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:51.810489 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 00:13:51.810552 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":154,"thread_start_us":145,"threads_started":1}
I20260812 00:13:51.811122 3289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08c2bdac31491cb781ce93f56425d4" candidate_uuid: "e6beb214562c4740beea5f35c4689f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" is_pre_election: true
I20260812 00:13:51.811123 3151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08c2bdac31491cb781ce93f56425d4" candidate_uuid: "e6beb214562c4740beea5f35c4689f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840b26cc3cb547cc9ce30a4e255c275b" is_pre_election: true
I20260812 00:13:51.811299 3289 raft_consensus.cc:2454] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6beb214562c4740beea5f35c4689f86 in term 0.
I20260812 00:13:51.811299 3151 raft_consensus.cc:2454] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6beb214562c4740beea5f35c4689f86 in term 0.
I20260812 00:13:51.811561 3360 leader_election.cc:302] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 840b26cc3cb547cc9ce30a4e255c275b, e6beb214562c4740beea5f35c4689f86; no voters:
I20260812 00:13:51.811754 3501 raft_consensus.cc:2781] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:51.811817 3501 raft_consensus.cc:482] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:51.811842 3501 raft_consensus.cc:3037] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:51.812674 3501 raft_consensus.cc:504] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.812825 3501 leader_election.cc:288] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [CANDIDATE]: Term 1 election: Requested vote from peers 4ac8fb7cfdc94f58b31f6e59ca5b7449 (127.16.99.194:41225), 840b26cc3cb547cc9ce30a4e255c275b (127.16.99.193:45005)
I20260812 00:13:51.812995 3151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08c2bdac31491cb781ce93f56425d4" candidate_uuid: "e6beb214562c4740beea5f35c4689f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840b26cc3cb547cc9ce30a4e255c275b"
I20260812 00:13:51.813099 3151 raft_consensus.cc:3037] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:51.813016 3289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee08c2bdac31491cb781ce93f56425d4" candidate_uuid: "e6beb214562c4740beea5f35c4689f86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449"
I20260812 00:13:51.813277 3289 raft_consensus.cc:3037] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:51.814007 3151 raft_consensus.cc:2454] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6beb214562c4740beea5f35c4689f86 in term 1.
I20260812 00:13:51.814184 3360 leader_election.cc:302] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 840b26cc3cb547cc9ce30a4e255c275b, e6beb214562c4740beea5f35c4689f86; no voters:
I20260812 00:13:51.814206 3289 raft_consensus.cc:2454] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6beb214562c4740beea5f35c4689f86 in term 1.
I20260812 00:13:51.814291 3501 raft_consensus.cc:2781] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:51.814499 3501 raft_consensus.cc:686] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 1 LEADER]: Becoming Leader. State: Replica: e6beb214562c4740beea5f35c4689f86, State: Running, Role: LEADER
I20260812 00:13:51.814641 3501 consensus_queue.cc:237] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } }
I20260812 00:13:51.816045 2995 catalog_manager.cc:5495] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6beb214562c4740beea5f35c4689f86 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "e6beb214562c4740beea5f35c4689f86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6beb214562c4740beea5f35c4689f86" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:51.836124 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.833205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.16.99.195:42619)
0812 00:13:51.833275 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:51.833277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:51.833286 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:13:51.833349 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:51.833350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:51.833370 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:51.833436 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:51.833592 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.834114 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.835078 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:51.835280 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:51.835344 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:51.835574 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:51.835703 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:51.835799 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:51.835916 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 00:13:51.835939 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:13:51.838366 3151 raft_consensus.cc:1261] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 1 FOLLOWER]: Refusing update from remote peer e6beb214562c4740beea5f35c4689f86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:51.838428 3289 raft_consensus.cc:1261] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 1 FOLLOWER]: Refusing update from remote peer e6beb214562c4740beea5f35c4689f86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:51.838780 3501 consensus_queue.cc:1035] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "840b26cc3cb547cc9ce30a4e255c275b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 45005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:51.838989 3504 consensus_queue.cc:1035] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac8fb7cfdc94f58b31f6e59ca5b7449" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 41225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:51.842298 3507 mvcc.cc:205] Tried to move back new op lower bound from 7317477916006469632 to 7317477915913560064. Current Snapshot: MvccSnapshot[applied={T|T < 7317477916006469632}]
I20260812 00:13:51.842317 3510 mvcc.cc:205] Tried to move back new op lower bound from 7317477916006469632 to 7317477915913560064. Current Snapshot: MvccSnapshot[applied={T|T < 7317477916006469632}]
I20260812 00:13:51.845116 2996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:51.845250 2996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:51.846149 2996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 888b80765fa74788924f8b4c06b401a7: Sending DeleteTablet for 3 replicas of tablet ee08c2bdac31491cb781ce93f56425d4
I20260812 00:13:51.846572 3127 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08c2bdac31491cb781ce93f56425d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:51 UTC) from {username='slave'} at 127.0.0.1:52180
I20260812 00:13:51.846623 3407 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08c2bdac31491cb781ce93f56425d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:51 UTC) from {username='slave'} at 127.0.0.1:48986
I20260812 00:13:51.846714 3267 tablet_service.cc:1514] Processing DeleteTablet for tablet ee08c2bdac31491cb781ce93f56425d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:51 UTC) from {username='slave'} at 127.0.0.1:43254
I20260812 00:13:51.846899 3519 tablet_replica.cc:330] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b: stopping tablet replica
I20260812 00:13:51.847018 3519 raft_consensus.cc:2229] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:51.847105 3521 tablet_replica.cc:330] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: stopping tablet replica
I20260812 00:13:51.847126 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 3063
I20260812 00:13:51.847152 3519 raft_consensus.cc:2258] T ee08c2bdac31491cb781ce93f56425d4 P 840b26cc3cb547cc9ce30a4e255c275b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:51.847203 3521 raft_consensus.cc:2229] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:51.847304 3521 raft_consensus.cc:2258] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:51.847841 3521 ts_tablet_manager.cc:1905] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:51.849293 3520 tablet_replica.cc:330] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: stopping tablet replica
I20260812 00:13:51.849414 3520 raft_consensus.cc:2229] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:13:51.849503 3521 ts_tablet_manager.cc:1918] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:51.849570 3521 log.cc:1196] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/ee08c2bdac31491cb781ce93f56425d4
I20260812 00:13:51.849566 3520 raft_consensus.cc:2258] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:51.849879 3521 ts_tablet_manager.cc:1939] T ee08c2bdac31491cb781ce93f56425d4 P 4ac8fb7cfdc94f58b31f6e59ca5b7449: Deleting consensus metadata
I20260812 00:13:51.850129 3520 ts_tablet_manager.cc:1905] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:51.850387 2980 catalog_manager.cc:4841] TS 4ac8fb7cfdc94f58b31f6e59ca5b7449 (127.16.99.194:41225): tablet ee08c2bdac31491cb781ce93f56425d4 (table test-table [id=9c5bd0183ab846c68719844cf1c7b3bd]) successfully deleted
I20260812 00:13:51.851677 3520 ts_tablet_manager.cc:1918] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:51.851750 3520 log.cc:1196] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/ee08c2bdac31491cb781ce93f56425d4
I20260812 00:13:51.851955 3520 ts_tablet_manager.cc:1939] T ee08c2bdac31491cb781ce93f56425d4 P e6beb214562c4740beea5f35c4689f86: Deleting consensus metadata
I20260812 00:13:51.852365 2980 catalog_manager.cc:4841] TS e6beb214562c4740beea5f35c4689f86 (127.16.99.195:42619): tablet ee08c2bdac31491cb781ce93f56425d4 (table test-table [id=9c5bd0183ab846c68719844cf1c7b3bd]) successfully deleted
W20260812 00:13:51.856726 2983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:45005 (error 108)
I20260812 00:13:51.857002 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 3202
I20260812 00:13:51.858446 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:51.857148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:45005 (local address 127.0.0.1:52182)
0812 00:13:51.858296 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:51.858397 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1121}
W20260812 00:13:51.858589 2983 catalog_manager.cc:4584] TS 840b26cc3cb547cc9ce30a4e255c275b (127.16.99.193:45005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee08c2bdac31491cb781ce93f56425d4: Network error: Client connection negotiation failed: client connection to 127.16.99.193:45005: connect: Connection refused (error 111)
I20260812 00:13:51.866725 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 3341
I20260812 00:13:51.873099 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 2964
2026-08-12T00:13:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (855 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:13:51 dist-test-slave-zpfg krb5kdc[3525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:13:51 dist-test-slave-zpfg krb5kdc[3525](info): set up 2 sockets
Aug 12 00:13:51 dist-test-slave-zpfg krb5kdc[3525](info): commencing operation
krb5kdc: starting...
W20260812 00:13:53.003861 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:13:53.145995 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40085
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:40085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:53.234863 3538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:53.235142 3538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:53.235189 3538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:53.236480 3538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:53.236536 3538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:53.236550 3538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:53.236563 3538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:53.236575 3538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:53.238735 3538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:40085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:53.239063 3538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:53.239355 3538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:53.245357 3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.245381 3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.245397 3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:53.245625 3538 server_base.cc:1034] running on GCE node
I20260812 00:13:53.245931 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:53.246217 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:53.247443 3538 hybrid_clock.cc:648] HybridClock initialized: now 1786493633247421 us; error 34 us; skew 500 ppm
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:53.250567 3538 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:13:53.251113 3538 webserver.cc:460] Webserver started at http://127.16.99.254:36381/ using document root <none> and password file <none>
I20260812 00:13:53.251391 3538 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:53.251467 3538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:53.251711 3538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:53.252919 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "8f6ad26b4e9d4df68a9f6a107050cb78"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "c917b5d4e7ead1a7efa4c8eb0ec85702"
server_key_iv: "366b7dd37487031e5b248bed411588b8"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.253315 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "8f6ad26b4e9d4df68a9f6a107050cb78"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "c917b5d4e7ead1a7efa4c8eb0ec85702"
server_key_iv: "366b7dd37487031e5b248bed411588b8"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.254915 3538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:13:53.255918 3553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.256278 3538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:53.256421 3538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "8f6ad26b4e9d4df68a9f6a107050cb78"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "c917b5d4e7ead1a7efa4c8eb0ec85702"
server_key_iv: "366b7dd37487031e5b248bed411588b8"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.256515 3538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:53.280946 3538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:53.284274 3538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:53.284487 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:53.289296 3538 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:40085
I20260812 00:13:53.289336 3610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:40085 every 8 connection(s)
I20260812 00:13:53.289748 3538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:53.290885 3611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:53.292738 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 3538
I20260812 00:13:53.292889 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:13:53.293187 16783 external_mini_cluster.cc:1428] Setting key e33d9ffecdc0fb8dc58ee2c124e27d28
I20260812 00:13:53.293605 3611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78: Bootstrap starting.
I20260812 00:13:53.294415 3611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:53.294775 3611 log.cc:824] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:53.295739 3611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78: No bootstrap required, opened a new log
I20260812 00:13:53.298941 3611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40085 } }
I20260812 00:13:53.299154 3611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:53.299212 3611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6ad26b4e9d4df68a9f6a107050cb78, State: Initialized, Role: FOLLOWER
I20260812 00:13:53.299431 3611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40085 } }
I20260812 00:13:53.299551 3611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:53.299592 3611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:53.299649 3611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:53.300601 3611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40085 } }
I20260812 00:13:53.300773 3611 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6ad26b4e9d4df68a9f6a107050cb78; no voters:
I20260812 00:13:53.301004 3611 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:53.301067 3616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:53.301319 3611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:53.301350 3616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6ad26b4e9d4df68a9f6a107050cb78, State: Running, Role: LEADER
I20260812 00:13:53.301607 3616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40085 } }
I20260812 00:13:53.302035 3617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40085 } } }
I20260812 00:13:53.302126 3617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:53.302314 3617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6ad26b4e9d4df68a9f6a107050cb78. Latest consensus state: current_term: 1 leader_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6ad26b4e9d4df68a9f6a107050cb78" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40085 } } }
I20260812 00:13:53.302405 3617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:53.302462 3625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:53.303078 3625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:53.306262 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.293840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.16.99.254:40085)
0812 00:13:53.294067 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.294080 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.294179 (+ 99us) server_negotiation.cc:407] Connection header received
0812 00:13:53.294918 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.294930 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.295004 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.295188 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:53.295846 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.296789 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.298035 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.298633 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.298785 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.302049 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:53.302053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:53.302053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:53.302067 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:53.304401 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.304836 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.304839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.304840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.304967 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.305195 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.305197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.305197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.305577 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:53.305786 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.305867 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.306028 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260812 00:13:53.307336 3625 catalog_manager.cc:1347] Generated new cluster ID: 412b141f6ba6465ea58bfc92762722f0
I20260812 00:13:53.307403 3625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:53.324700 3625 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:53.325582 3625 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:53.337397 3625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78: Generated new TSK 0
I20260812 00:13:53.337680 3625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:13:53.369916 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40085
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:53.463155 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:53.463382 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:53.463413 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:53.464579 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:53.464623 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:53.464653 3637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:53.466295 3637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:53.466568 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:53.466822 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:13:53.472554 3637 server_base.cc:1034] running on GCE node
W20260812 00:13:53.472570 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.472586 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.472589 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:53.473057 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:53.473366 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:53.474545 3637 hybrid_clock.cc:648] HybridClock initialized: now 1786493633474534 us; error 44 us; skew 500 ppm
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:53.477478 3637 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:13:53.478034 3637 webserver.cc:460] Webserver started at http://127.16.99.193:43349/ using document root <none> and password file <none>
I20260812 00:13:53.478298 3637 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:53.478397 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:53.478561 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:53.479701 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "777ed3fbc25888c3a021d71f42643e3f"
server_key_iv: "28e51963e2f7cda6a671f25396d39db5"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.480091 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "777ed3fbc25888c3a021d71f42643e3f"
server_key_iv: "28e51963e2f7cda6a671f25396d39db5"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.481503 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:13:53.482465 3653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.482645 3637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:53.482749 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "777ed3fbc25888c3a021d71f42643e3f"
server_key_iv: "28e51963e2f7cda6a671f25396d39db5"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.482841 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:53.499408 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:53.502053 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:53.502306 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:53.502720 3637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:53.503082 3637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:53.503136 3637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.503190 3637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:53.503221 3637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.509773 3637 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:46051
I20260812 00:13:53.509847 3772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:46051 every 8 connection(s)
I20260812 00:13:53.510180 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:53.516958 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 3637
I20260812 00:13:53.517095 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:13:53.517306 16783 external_mini_cluster.cc:1428] Setting key 5d54f9d1e872a2e98a0bfd35684e1415
I20260812 00:13:53.520859 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.511040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:36979 (local address 127.16.99.254:40085)
0812 00:13:53.511109 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.511111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.511409 (+ 298us) server_negotiation.cc:407] Connection header received
0812 00:13:53.511900 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.511901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.511911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.511975 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:53.512943 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.513514 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.514555 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.514811 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.514888 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.517484 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:53.517487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:53.517487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:53.517497 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:53.519247 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.519723 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.519725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.519725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.519803 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.520096 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.520097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.520097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.520225 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:53.520316 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.520640 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.520762 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:13:53.521061 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.510999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40085 (local address 127.16.99.193:36979)
0812 00:13:53.511186 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:53.511321 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:53.511817 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:53.512033 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:53.512037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:53.512640 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:53.512847 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.512852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.513605 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.513606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.514471 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.514473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.514907 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.514908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.515009 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.515556 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:53.515564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:53.517363 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:53.519350 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:53.519362 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:53.519364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:53.519634 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:53.519877 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:53.519878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:53.519879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:53.520017 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:53.520306 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:53.520315 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:53.520490 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 00:13:53.520607 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 00:13:53.520793 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
I20260812 00:13:53.521728 3773 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40085
I20260812 00:13:53.521839 3773 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:53.522056 3773 heartbeater.cc:511] Master 127.16.99.254:40085 requested a full tablet report, sending...
I20260812 00:13:53.522748 3570 ts_manager.cc:194] Registered new tserver with Master: 43546a5c6c6c4bd4bf3dc406ed0dab5b (127.16.99.193:46051)
I20260812 00:13:53.523779 3570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:36979
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:13:53.550174 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40085
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:53.641423 3778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:53.641641 3778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:53.641665 3778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:53.642793 3778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:53.642830 3778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:53.642858 3778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:53.644243 3778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:53.644444 3778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:53.644620 3778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:53.650228 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.650243 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.650235 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:53.650550 3778 server_base.cc:1034] running on GCE node
I20260812 00:13:53.650923 3778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:53.651229 3778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:53.652415 3778 hybrid_clock.cc:648] HybridClock initialized: now 1786493633652401 us; error 32 us; skew 500 ppm
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:53.655402 3778 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:13:53.655920 3778 webserver.cc:460] Webserver started at http://127.16.99.194:42593/ using document root <none> and password file <none>
I20260812 00:13:53.656210 3778 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:53.656291 3778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:53.656453 3778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:53.657667 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "7306dccbb05b443295ec5b97b2cc9539"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "349a713b6aa38597b487cab4dfc9f405"
server_key_iv: "71d7aa412079bf3eca736f640b4e8ce0"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.658080 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "7306dccbb05b443295ec5b97b2cc9539"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "349a713b6aa38597b487cab4dfc9f405"
server_key_iv: "71d7aa412079bf3eca736f640b4e8ce0"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.659790 3778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260812 00:13:53.660632 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.660851 3778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:53.660962 3778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "7306dccbb05b443295ec5b97b2cc9539"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "349a713b6aa38597b487cab4dfc9f405"
server_key_iv: "71d7aa412079bf3eca736f640b4e8ce0"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.661057 3778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:53.686095 3778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:53.688757 3778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:53.688961 3778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:53.689270 3778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:53.689626 3778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:53.689694 3778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.689733 3778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:53.689783 3778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.696738 3778 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:37437
I20260812 00:13:53.696801 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:37437 every 8 connection(s)
I20260812 00:13:53.697221 3778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:13:53.707546 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 3778
I20260812 00:13:53.707717 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:13:53.707939 16783 external_mini_cluster.cc:1428] Setting key 1eb05b114089afbd9eade09ef5e3de2f
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:53.713337 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.698302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:39817 (local address 127.16.99.254:40085)
0812 00:13:53.698377 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.698379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.698518 (+ 139us) server_negotiation.cc:407] Connection header received
0812 00:13:53.699052 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.699053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.699064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.699220 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:53.700321 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.700967 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.702223 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.702517 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.702581 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.709653 (+ 7072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:53.709655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:53.709655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:53.709662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:53.711625 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.712123 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.712124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.712124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.712193 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.712521 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.712522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.712523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.712692 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:53.712775 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.713065 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.713178 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 00:13:53.713356 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.698134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40085 (local address 127.16.99.194:39817)
0812 00:13:53.698360 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:53.698421 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:53.698958 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:53.699228 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:53.699233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:53.699960 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:53.700189 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.700196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.701121 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.701122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.702119 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.702121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.702604 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.702605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.702774 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.707657 (+ 4883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:53.707668 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:53.709524 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:53.711755 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:53.711763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:53.711765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:53.712034 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:53.712328 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:53.712328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:53.712329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:53.712446 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:53.712774 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:53.712780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:53.712917 (+ 137us) client_negotiation.cc:769] Sending connection context
0812 00:13:53.713013 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 00:13:53.713157 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":115,"threads_started":1}
I20260812 00:13:53.714061 3913 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40085
I20260812 00:13:53.714200 3913 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:53.714474 3913 heartbeater.cc:511] Master 127.16.99.254:40085 requested a full tablet report, sending...
I20260812 00:13:53.714973 3570 ts_manager.cc:194] Registered new tserver with Master: 7306dccbb05b443295ec5b97b2cc9539 (127.16.99.194:37437)
I20260812 00:13:53.715430 3570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:39817
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:13:53.742462 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40085
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:53.832414 3918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:53.832680 3918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:53.832732 3918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:53.833959 3918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:53.834038 3918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:53.834085 3918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:53.835729 3918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:53.836027 3918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:53.836290 3918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:53.842082 3923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.842082 3926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:53.842075 3924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:53.842270 3918 server_base.cc:1034] running on GCE node
I20260812 00:13:53.842782 3918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:53.843031 3918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:53.844210 3918 hybrid_clock.cc:648] HybridClock initialized: now 1786493633844175 us; error 47 us; skew 500 ppm
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:53.847689 3918 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:13:53.848266 3918 webserver.cc:460] Webserver started at http://127.16.99.195:40379/ using document root <none> and password file <none>
I20260812 00:13:53.848520 3918 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:53.848595 3918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:53.848775 3918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:53.850090 3918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "e2f05746aa1d401ca712a24f716657bb"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "641057d91896dcb5b77b44c587df825a"
server_key_iv: "2427dc001141c1d10342b2616569d83c"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.850534 3918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "e2f05746aa1d401ca712a24f716657bb"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "641057d91896dcb5b77b44c587df825a"
server_key_iv: "2427dc001141c1d10342b2616569d83c"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.852092 3918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:53.852938 3933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.853125 3918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:53.853245 3918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "e2f05746aa1d401ca712a24f716657bb"
format_stamp: "Formatted at 2026-08-12 00:13:53 on dist-test-slave-zpfg"
server_key: "641057d91896dcb5b77b44c587df825a"
server_key_iv: "2427dc001141c1d10342b2616569d83c"
server_key_version: "encryptionkey@0"
I20260812 00:13:53.853327 3918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:53.868592 3918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:53.870853 3918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:53.871059 3918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:53.871459 3918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:53.871852 3918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:53.871912 3918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.871956 3918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:53.871985 3918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:53.878902 3918 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:33695
I20260812 00:13:53.879019 4053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:33695 every 8 connection(s)
I20260812 00:13:53.879356 3918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:13:53 dist-test-slave-zpfg krb5kdc[3525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493633, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:53.888831 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 3918
I20260812 00:13:53.888952 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:13:53.889242 16783 external_mini_cluster.cc:1428] Setting key 4e3a7df332bcf69f9d516eefadf5a870
I20260812 00:13:53.889914 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.880168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:59985 (local address 127.16.99.254:40085)
0812 00:13:53.880239 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.880242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.880480 (+ 238us) server_negotiation.cc:407] Connection header received
0812 00:13:53.880988 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.880990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.881008 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.881074 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:53.882067 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.882704 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.883731 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.883938 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.884012 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.886389 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:53.886391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:53.886391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:53.886399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:53.888000 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.888430 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.888431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.888432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.888498 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.888889 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.888891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.888891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.889095 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:53.889181 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.889714 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.889810 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:13:53.890255 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.880163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:40085 (local address 127.16.99.195:59985)
0812 00:13:53.880352 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:53.880391 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:53.880875 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:53.881146 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:53.881150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:53.881756 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:53.881959 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.881963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.882795 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.882796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.883622 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.883623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.884027 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.884028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.884133 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.884751 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:53.884763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:53.886249 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:53.888087 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:53.888094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:53.888096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:53.888342 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:53.888615 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:53.888616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:53.888616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:53.888768 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:53.889273 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:53.889281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:53.889507 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 00:13:53.889675 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 00:13:53.889896 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260812 00:13:53.890786 4054 heartbeater.cc:347] Connected to a master server at 127.16.99.254:40085
I20260812 00:13:53.890883 4054 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:53.891060 4054 heartbeater.cc:511] Master 127.16.99.254:40085 requested a full tablet report, sending...
I20260812 00:13:53.891520 3570 ts_manager.cc:194] Registered new tserver with Master: e2f05746aa1d401ca712a24f716657bb (127.16.99.195:33695)
I20260812 00:13:53.891985 3570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:59985
I20260812 00:13:53.901862 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:53.911288 3614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.903568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.16.99.254:40085)
0812 00:13:53.903638 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.903640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.903878 (+ 238us) server_negotiation.cc:407] Connection header received
0812 00:13:53.903918 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.903919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.903931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.903986 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:53.904614 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.905125 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.906000 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.906275 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.906432 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.907986 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:53.907987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:53.907987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:53.907995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:53.909536 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.909995 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.909997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.909997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.910058 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:53.910461 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:53.910462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:53.910462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:53.910631 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:53.910733 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.911027 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.911149 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:13:53.913756 3570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:53.915858 3570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:53.923367 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.918821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:33695 (local address 127.0.0.1:43964)
0812 00:13:53.919034 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:53.919080 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:53.919163 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:53.919808 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:53.919809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:53.919810 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:53.919855 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.919856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.920809 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.920810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.922051 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.922054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.922995 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.922996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.923092 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.923107 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 00:13:53.923192 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:13:53.923257 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260812 00:13:53.923461 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.919322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.16.99.193:46051)
0812 00:13:53.919486 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.919491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.919502 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:53.919543 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.919545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.919561 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.919691 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:53.919792 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.921061 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.922331 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.923082 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.923164 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.923217 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.923330 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.923371 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260812 00:13:53.923769 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.919000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.16.99.194:37437)
0812 00:13:53.919211 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.919216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.919230 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:13:53.919351 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.919354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.919373 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.919475 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:53.919934 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.921115 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.922533 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.923268 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.923335 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.923382 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.923636 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.923686 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260812 00:13:53.923477 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.918864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46051 (local address 127.0.0.1:38598)
0812 00:13:53.919280 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:53.919291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:53.919339 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:53.919691 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:53.919692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:53.919694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:53.919740 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.919742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.921191 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.921192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.922248 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.922250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.923182 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.923183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.923243 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.923247 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:53.923312 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:13:53.923351 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":114,"threads_started":1}
I20260812 00:13:53.923874 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.919111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.16.99.195:33695)
0812 00:13:53.919599 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:53.919604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:53.919615 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:53.919653 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:53.919656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:53.919671 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:53.919927 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:53.919989 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.920683 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.922163 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.922900 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.922998 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.923069 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:53.923751 (+ 682us) server_negotiation.cc:299] Negotiation successful
0812 00:13:53.923791 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":96,"threads_started":1}
I20260812 00:13:53.923667 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:53.918824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:37437 (local address 127.0.0.1:35922)
0812 00:13:53.919160 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:53.919173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:53.919235 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:53.919500 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:53.919504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:53.919507 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:53.919597 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.919599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.921221 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.921223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.922447 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:53.922450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:53.923420 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:53.923421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:53.923486 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:53.923491 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:53.923551 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 00:13:53.923594 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":86,"threads_started":1}
I20260812 00:13:53.924875 3983 tablet_service.cc:1467] Processing CreateTablet for tablet e4cf10224df546d281c8243b1acac603 (DEFAULT_TABLE table=test-table [id=2be5db151bab4d4dbf8b7941b3af050a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:53.924875 3703 tablet_service.cc:1467] Processing CreateTablet for tablet e4cf10224df546d281c8243b1acac603 (DEFAULT_TABLE table=test-table [id=2be5db151bab4d4dbf8b7941b3af050a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:53.925060 3844 tablet_service.cc:1467] Processing CreateTablet for tablet e4cf10224df546d281c8243b1acac603 (DEFAULT_TABLE table=test-table [id=2be5db151bab4d4dbf8b7941b3af050a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:53.925297 3703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cf10224df546d281c8243b1acac603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:53.925303 3983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cf10224df546d281c8243b1acac603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:53.925402 3844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cf10224df546d281c8243b1acac603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:53.928504 4071 tablet_bootstrap.cc:492] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Bootstrap starting.
I20260812 00:13:53.928845 4072 tablet_bootstrap.cc:492] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: Bootstrap starting.
I20260812 00:13:53.929064 4073 tablet_bootstrap.cc:492] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Bootstrap starting.
I20260812 00:13:53.929770 4073 tablet_bootstrap.cc:654] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:53.929797 4072 tablet_bootstrap.cc:654] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:53.929797 4071 tablet_bootstrap.cc:654] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:53.930332 4073 log.cc:824] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:53.930533 4072 log.cc:824] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:53.930533 4071 log.cc:824] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:53.931341 4073 tablet_bootstrap.cc:492] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: No bootstrap required, opened a new log
I20260812 00:13:53.931362 4071 tablet_bootstrap.cc:492] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: No bootstrap required, opened a new log
I20260812 00:13:53.931434 4073 ts_tablet_manager.cc:1397] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:13:53.931434 4071 ts_tablet_manager.cc:1397] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:13:53.931475 4072 tablet_bootstrap.cc:492] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: No bootstrap required, opened a new log
I20260812 00:13:53.931545 4072 ts_tablet_manager.cc:1397] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:13:53.933133 4072 raft_consensus.cc:348] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:53.933328 4072 raft_consensus.cc:374] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:53.933375 4072 raft_consensus.cc:729] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43546a5c6c6c4bd4bf3dc406ed0dab5b, State: Initialized, Role: FOLLOWER
I20260812 00:13:53.933534 4072 consensus_queue.cc:260] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:53.933703 4073 raft_consensus.cc:348] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:53.933703 4071 raft_consensus.cc:348] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:53.933853 3773 heartbeater.cc:503] Master 127.16.99.254:40085 was elected leader, sending a full tablet report...
I20260812 00:13:53.933864 4072 ts_tablet_manager.cc:1428] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:13:53.933899 4073 raft_consensus.cc:374] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:53.933899 4071 raft_consensus.cc:374] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:53.933940 4071 raft_consensus.cc:729] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f05746aa1d401ca712a24f716657bb, State: Initialized, Role: FOLLOWER
I20260812 00:13:53.933941 4073 raft_consensus.cc:729] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7306dccbb05b443295ec5b97b2cc9539, State: Initialized, Role: FOLLOWER
I20260812 00:13:53.934067 4073 consensus_queue.cc:260] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:53.934067 4071 consensus_queue.cc:260] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:53.934425 4073 ts_tablet_manager.cc:1428] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:13:53.934545 4071 ts_tablet_manager.cc:1428] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:13:53.934597 4054 heartbeater.cc:503] Master 127.16.99.254:40085 was elected leader, sending a full tablet report...
I20260812 00:13:53.934816 3913 heartbeater.cc:503] Master 127.16.99.254:40085 was elected leader, sending a full tablet report...
W20260812 00:13:53.947902 3914 tablet.cc:2369] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:13:54.011009 3774 tablet.cc:2369] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:13:54.130060 4055 tablet.cc:2369] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:54.172295 4078 raft_consensus.cc:482] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:54.172444 4078 raft_consensus.cc:504] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:54.173002 4078 leader_election.cc:288] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f05746aa1d401ca712a24f716657bb (127.16.99.195:33695), 43546a5c6c6c4bd4bf3dc406ed0dab5b (127.16.99.193:46051)
I20260812 00:13:54.177908 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.173358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46051 (local address 127.16.99.194:49989)
0812 00:13:54.173435 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.173451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.173529 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.173849 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.173850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.173852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:54.173975 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.173978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.175244 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.175245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.176675 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.176679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.177538 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.177539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.177678 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.177684 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.177750 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.177809 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":106}
I20260812 00:13:54.177937 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.173463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:49989 (local address 127.16.99.193:46051)
0812 00:13:54.173511 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.173514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.173527 (+ 13us) server_negotiation.cc:407] Connection header received
0812 00:13:54.173723 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.173725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.173747 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.173820 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:54.174081 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.175007 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.176810 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.177435 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.177529 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.177539 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.177799 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.177856 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:13:54.177937 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.173499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:32815 (local address 127.16.99.195:33695)
0812 00:13:54.173568 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.173570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.173722 (+ 152us) server_negotiation.cc:407] Connection header received
0812 00:13:54.173760 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.173761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.173772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.173828 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:54.174166 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.175007 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.176810 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.177580 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.177664 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.177672 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.177830 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.177870 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 00:13:54.177914 4080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.173360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:33695 (local address 127.16.99.194:32815)
0812 00:13:54.173594 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.173606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.173671 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.173849 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.173850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.173852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:54.173992 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.173994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.175244 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.175245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.176675 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.176679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.177670 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.177670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.177744 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.177748 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.177802 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.177833 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":151,"thread_start_us":89,"threads_started":1}
I20260812 00:13:54.178473 4004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cf10224df546d281c8243b1acac603" candidate_uuid: "7306dccbb05b443295ec5b97b2cc9539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f05746aa1d401ca712a24f716657bb" is_pre_election: true
I20260812 00:13:54.178534 3723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cf10224df546d281c8243b1acac603" candidate_uuid: "7306dccbb05b443295ec5b97b2cc9539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" is_pre_election: true
I20260812 00:13:54.178702 4004 raft_consensus.cc:2454] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7306dccbb05b443295ec5b97b2cc9539 in term 0.
I20260812 00:13:54.178731 3723 raft_consensus.cc:2454] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7306dccbb05b443295ec5b97b2cc9539 in term 0.
I20260812 00:13:54.179008 3796 leader_election.cc:302] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43546a5c6c6c4bd4bf3dc406ed0dab5b, 7306dccbb05b443295ec5b97b2cc9539; no voters:
I20260812 00:13:54.179212 4078 raft_consensus.cc:2781] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:54.179278 4078 raft_consensus.cc:482] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:54.179297 4078 raft_consensus.cc:3037] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:54.180150 4078 raft_consensus.cc:504] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:54.180323 4078 leader_election.cc:288] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [CANDIDATE]: Term 1 election: Requested vote from peers e2f05746aa1d401ca712a24f716657bb (127.16.99.195:33695), 43546a5c6c6c4bd4bf3dc406ed0dab5b (127.16.99.193:46051)
I20260812 00:13:54.180536 4004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cf10224df546d281c8243b1acac603" candidate_uuid: "7306dccbb05b443295ec5b97b2cc9539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f05746aa1d401ca712a24f716657bb"
I20260812 00:13:54.180622 3723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cf10224df546d281c8243b1acac603" candidate_uuid: "7306dccbb05b443295ec5b97b2cc9539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b"
I20260812 00:13:54.180630 4004 raft_consensus.cc:3037] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:54.180732 3723 raft_consensus.cc:3037] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:54.181530 4004 raft_consensus.cc:2454] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7306dccbb05b443295ec5b97b2cc9539 in term 1.
I20260812 00:13:54.181703 3723 raft_consensus.cc:2454] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7306dccbb05b443295ec5b97b2cc9539 in term 1.
I20260812 00:13:54.181761 3795 leader_election.cc:302] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7306dccbb05b443295ec5b97b2cc9539, e2f05746aa1d401ca712a24f716657bb; no voters:
I20260812 00:13:54.181883 4078 raft_consensus.cc:2781] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:54.182102 4078 raft_consensus.cc:686] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 1 LEADER]: Becoming Leader. State: Replica: 7306dccbb05b443295ec5b97b2cc9539, State: Running, Role: LEADER
I20260812 00:13:54.182237 4078 consensus_queue.cc:237] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } }
I20260812 00:13:54.183511 3570 catalog_manager.cc:5495] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7306dccbb05b443295ec5b97b2cc9539 (127.16.99.194). New cstate: current_term: 1 leader_uuid: "7306dccbb05b443295ec5b97b2cc9539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7306dccbb05b443295ec5b97b2cc9539" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 37437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:54.213974 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.210661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.16.99.194:37437)
0812 00:13:54.210737 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.210739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.210756 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:13:54.210854 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.210856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.210884 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.210959 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:54.211116 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.211710 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.212811 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.213088 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.213172 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.213469 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:54.213580 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:54.213712 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.213787 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.213801 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:13:54.216657 3723 raft_consensus.cc:1261] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 1 FOLLOWER]: Refusing update from remote peer 7306dccbb05b443295ec5b97b2cc9539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:54.216657 4004 raft_consensus.cc:1261] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 1 FOLLOWER]: Refusing update from remote peer 7306dccbb05b443295ec5b97b2cc9539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:54.217129 4078 consensus_queue.cc:1035] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f05746aa1d401ca712a24f716657bb" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:54.217337 4081 consensus_queue.cc:1035] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43546a5c6c6c4bd4bf3dc406ed0dab5b" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:54.220222 4084 mvcc.cc:205] Tried to move back new op lower bound from 7317477925747716096 to 7317477925611425792. Current Snapshot: MvccSnapshot[applied={T|T < 7317477925747716096}]
I20260812 00:13:54.220919 4086 mvcc.cc:205] Tried to move back new op lower bound from 7317477925747716096 to 7317477925611425792. Current Snapshot: MvccSnapshot[applied={T|T < 7317477925747716096}]
I20260812 00:13:54.223901 3570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:54.224009 3570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:54.224876 3570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f6ad26b4e9d4df68a9f6a107050cb78: Sending DeleteTablet for 3 replicas of tablet e4cf10224df546d281c8243b1acac603
I20260812 00:13:54.225317 3703 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cf10224df546d281c8243b1acac603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:54 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:38598
I20260812 00:13:54.225391 3983 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cf10224df546d281c8243b1acac603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:54 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:43964
I20260812 00:13:54.225432 3844 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cf10224df546d281c8243b1acac603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:54 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:35922
I20260812 00:13:54.225673 4097 tablet_replica.cc:330] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: stopping tablet replica
I20260812 00:13:54.225804 4096 tablet_replica.cc:330] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b: stopping tablet replica
I20260812 00:13:54.225832 4098 tablet_replica.cc:330] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: stopping tablet replica
I20260812 00:13:54.225946 4098 raft_consensus.cc:2229] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:54.226006 4096 raft_consensus.cc:2229] T e4cf10224df546d281c8243b1acac603 P 43546a5c6c6c4bd4bf3dc406ed0dab5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:54.226079 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 3637
I20260812 00:13:54.228080 4097 raft_consensus.cc:2229] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:13:54.228363 4098 raft_consensus.cc:2258] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:54.228654 4097 raft_consensus.cc:2258] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:54.229522 4098 ts_tablet_manager.cc:1905] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:54.231889 4098 ts_tablet_manager.cc:1918] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:54.232053 4098 log.cc:1196] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/e4cf10224df546d281c8243b1acac603
I20260812 00:13:54.232491 4098 ts_tablet_manager.cc:1939] T e4cf10224df546d281c8243b1acac603 P e2f05746aa1d401ca712a24f716657bb: Deleting consensus metadata
I20260812 00:13:54.233175 3554 catalog_manager.cc:4841] TS e2f05746aa1d401ca712a24f716657bb (127.16.99.195:33695): tablet e4cf10224df546d281c8243b1acac603 (table test-table [id=2be5db151bab4d4dbf8b7941b3af050a]) successfully deleted
I20260812 00:13:54.234913 4097 ts_tablet_manager.cc:1905] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:54.237655 4097 ts_tablet_manager.cc:1918] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:54.237779 4097 log.cc:1196] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/e4cf10224df546d281c8243b1acac603
I20260812 00:13:54.238193 4097 ts_tablet_manager.cc:1939] T e4cf10224df546d281c8243b1acac603 P 7306dccbb05b443295ec5b97b2cc9539: Deleting consensus metadata
I20260812 00:13:54.238905 3557 catalog_manager.cc:4841] TS 7306dccbb05b443295ec5b97b2cc9539 (127.16.99.194:37437): tablet e4cf10224df546d281c8243b1acac603 (table test-table [id=2be5db151bab4d4dbf8b7941b3af050a]) successfully deleted
W20260812 00:13:54.240159 3555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:46051 (error 108)
I20260812 00:13:54.240428 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 3778
I20260812 00:13:54.242209 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.240710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46051 (local address 127.0.0.1:38606)
0812 00:13:54.242123 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.242175 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1387}
W20260812 00:13:54.242396 3555 catalog_manager.cc:4584] TS 43546a5c6c6c4bd4bf3dc406ed0dab5b (127.16.99.193:46051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4cf10224df546d281c8243b1acac603: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46051: connect: Connection refused (error 111)
I20260812 00:13:54.248057 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 3918
I20260812 00:13:54.254784 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 3538
2026-08-12T00:13:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2383 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T00:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:54Z Disabled control of system clock
I20260812 00:13:54.287895 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38251
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:38251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:54.377152 4104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:54.377410 4104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:54.377434 4104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:54.378836 4104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:54.378895 4104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:54.378909 4104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:54.378923 4104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:54.378959 4104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:54.380474 4104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:38251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:54.380728 4104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:54.381006 4104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:54.386497 4112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:54.386521 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:54.386497 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:54.387213 4104 server_base.cc:1034] running on GCE node
I20260812 00:13:54.387502 4104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:54.387843 4104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:54.389045 4104 hybrid_clock.cc:648] HybridClock initialized: now 1786493634389029 us; error 34 us; skew 500 ppm
I20260812 00:13:54.390707 4104 webserver.cc:460] Webserver started at http://127.16.99.254:45773/ using document root <none> and password file <none>
I20260812 00:13:54.390998 4104 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:54.391083 4104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:54.391233 4104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:54.392393 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "17b8915989724011afc81da8b86cf4e5"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "a6fa4870b2b7a93227bd01da8f596cc1"
server_key_iv: "7cf07fd92a03fabda45a7f88cd4b7a5f"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.392799 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "17b8915989724011afc81da8b86cf4e5"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "a6fa4870b2b7a93227bd01da8f596cc1"
server_key_iv: "7cf07fd92a03fabda45a7f88cd4b7a5f"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.394433 4104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:13:54.395483 4118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.395800 4104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:54.395948 4104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "17b8915989724011afc81da8b86cf4e5"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "a6fa4870b2b7a93227bd01da8f596cc1"
server_key_iv: "7cf07fd92a03fabda45a7f88cd4b7a5f"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.396045 4104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:54.407785 4104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:54.408200 4104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:54.408387 4104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:54.413309 4104 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:38251
I20260812 00:13:54.413355 4174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:38251 every 8 connection(s)
I20260812 00:13:54.413765 4104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:54.415017 4175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:54.417661 4175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5: Bootstrap starting.
I20260812 00:13:54.418421 4175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:54.418738 4175 log.cc:824] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:54.419550 4175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5: No bootstrap required, opened a new log
I20260812 00:13:54.421416 4175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b8915989724011afc81da8b86cf4e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38251 } }
I20260812 00:13:54.421590 4175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:54.421617 4175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b8915989724011afc81da8b86cf4e5, State: Initialized, Role: FOLLOWER
I20260812 00:13:54.421795 4175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b8915989724011afc81da8b86cf4e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38251 } }
I20260812 00:13:54.421892 4175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:54.421919 4175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:54.421952 4175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:54.422667 4175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b8915989724011afc81da8b86cf4e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38251 } }
I20260812 00:13:54.422780 4175 leader_election.cc:302] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17b8915989724011afc81da8b86cf4e5; no voters:
I20260812 00:13:54.422935 4175 leader_election.cc:288] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:54.423019 4178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:54.423233 4178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [term 1 LEADER]: Becoming Leader. State: Replica: 17b8915989724011afc81da8b86cf4e5, State: Running, Role: LEADER
I20260812 00:13:54.423322 4175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:54.423377 4178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b8915989724011afc81da8b86cf4e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38251 } }
I20260812 00:13:54.423911 4180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17b8915989724011afc81da8b86cf4e5. Latest consensus state: current_term: 1 leader_uuid: "17b8915989724011afc81da8b86cf4e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b8915989724011afc81da8b86cf4e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38251 } } }
I20260812 00:13:54.423949 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 4104
I20260812 00:13:54.423995 4179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17b8915989724011afc81da8b86cf4e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b8915989724011afc81da8b86cf4e5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38251 } } }
I20260812 00:13:54.424018 4180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:54.424050 4179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:54.424073 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:13:54.424336 16783 external_mini_cluster.cc:1428] Setting key 8cd0625a989d83180d972bf0a57346eb
I20260812 00:13:54.424321 4187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:54.424906 4187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:54.426841 4187 catalog_manager.cc:1347] Generated new cluster ID: 7c3d30d18dc14f68bab384acf1fe917c
I20260812 00:13:54.426919 4187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:54.435412 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.425106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.16.99.254:38251)
0812 00:13:54.425882 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.425891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.425926 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:13:54.426066 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.426074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.426157 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.426332 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:54.426615 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.431378 (+ 4763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.432824 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.433535 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.433678 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.434606 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:54.434610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:54.434611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:54.434843 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:54.434888 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:54.434903 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:54.434975 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.435088 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.435176 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":502,"threads_started":1}
I20260812 00:13:54.447690 4187 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:54.448348 4187 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:54.454979 4187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5: Generated new TSK 0
I20260812 00:13:54.455224 4187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:13:54.459592 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38251
--builtin_ntp_servers=127.16.99.212:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:54.547596 4200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:54.547837 4200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:54.547859 4200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:54.549105 4200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:54.549160 4200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:54.549190 4200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:54.550722 4200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:54.551007 4200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:54.551343 4200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:54.557334 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:54.557312 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:54.557315 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:54.557780 4200 server_base.cc:1034] running on GCE node
I20260812 00:13:54.558118 4200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:54.558475 4200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:54.559691 4200 hybrid_clock.cc:648] HybridClock initialized: now 1786493634559663 us; error 60 us; skew 500 ppm
I20260812 00:13:54.561344 4200 webserver.cc:460] Webserver started at http://127.16.99.193:37105/ using document root <none> and password file <none>
I20260812 00:13:54.561622 4200 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:54.561689 4200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:54.561823 4200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:54.563339 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "4c6e110431fa4e7b81642724729d1031"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "584125a2f97d9a4997b8ef202acfa879"
server_key_iv: "f0e4cd2b4840125d7d816d1985579943"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.563764 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "4c6e110431fa4e7b81642724729d1031"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "584125a2f97d9a4997b8ef202acfa879"
server_key_iv: "f0e4cd2b4840125d7d816d1985579943"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.565562 4200 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 00:13:54.566643 4214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.566843 4200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:54.566949 4200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "4c6e110431fa4e7b81642724729d1031"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "584125a2f97d9a4997b8ef202acfa879"
server_key_iv: "f0e4cd2b4840125d7d816d1985579943"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.567034 4200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:54.578255 4200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:54.578675 4200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:54.578840 4200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:54.579106 4200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:54.579463 4200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:54.579500 4200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.579522 4200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:54.579568 4200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.586241 4200 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:42177
I20260812 00:13:54.586407 4331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:42177 every 8 connection(s)
I20260812 00:13:54.586736 4200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:54.592998 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.587661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:38515 (local address 127.16.99.254:38251)
0812 00:13:54.587755 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.587758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.588054 (+ 296us) server_negotiation.cc:407] Connection header received
0812 00:13:54.588848 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.588850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.588863 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.588920 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:54.589711 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.590283 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.591370 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.591599 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.591731 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.592645 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:54.592647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:54.592647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:54.592653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:54.592669 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:54.592680 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:54.592756 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.592855 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.592902 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 00:13:54.593205 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.587619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:38251 (local address 127.16.99.193:38515)
0812 00:13:54.587906 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.587952 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.588713 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.589014 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.589018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.589029 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:54.589612 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.589618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.590418 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.590419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.591271 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.591274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.591731 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.591732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.591889 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.592493 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:54.592493 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:54.592523 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:54.592548 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:54.592761 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:54.592767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:54.592771 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.592861 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.592977 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":160,"threads_started":1}
I20260812 00:13:54.593869 4332 heartbeater.cc:347] Connected to a master server at 127.16.99.254:38251
I20260812 00:13:54.593969 4332 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:54.594128 4332 heartbeater.cc:511] Master 127.16.99.254:38251 requested a full tablet report, sending...
I20260812 00:13:54.594707 4138 ts_manager.cc:194] Registered new tserver with Master: 4c6e110431fa4e7b81642724729d1031 (127.16.99.193:42177)
I20260812 00:13:54.595414 4138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.193:38515
I20260812 00:13:54.595780 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 4200
I20260812 00:13:54.595886 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:13:54.596029 16783 external_mini_cluster.cc:1428] Setting key 726b0f88d357b063bd92c50a00e58253
I20260812 00:13:54.597224 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38251
--builtin_ntp_servers=127.16.99.212:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:54.685726 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:54.685971 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:54.685995 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:54.687417 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:54.687477 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:54.687506 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:54.688956 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:54.689180 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:54.689477 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:54.695382 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:54.695372 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:54.695351 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:54.695636 4335 server_base.cc:1034] running on GCE node
I20260812 00:13:54.695901 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:54.696198 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:54.697427 4335 hybrid_clock.cc:648] HybridClock initialized: now 1786493634697377 us; error 64 us; skew 500 ppm
I20260812 00:13:54.699077 4335 webserver.cc:460] Webserver started at http://127.16.99.194:36907/ using document root <none> and password file <none>
I20260812 00:13:54.699482 4335 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:54.699568 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:54.699755 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:54.700968 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "60dcb845145c1c9b16d3e4772f14d425"
server_key_iv: "1a0186de0acc2b1161ffbb48707996f5"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.701383 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "60dcb845145c1c9b16d3e4772f14d425"
server_key_iv: "1a0186de0acc2b1161ffbb48707996f5"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.703028 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260812 00:13:54.704385 4350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.704615 4335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:54.704727 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "60dcb845145c1c9b16d3e4772f14d425"
server_key_iv: "1a0186de0acc2b1161ffbb48707996f5"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.704826 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:54.727335 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:54.727735 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:54.727909 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:54.728199 4335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:54.728587 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:54.728652 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.728698 4335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:54.728729 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.735532 4335 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:44245
I20260812 00:13:54.735579 4469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:44245 every 8 connection(s)
I20260812 00:13:54.735981 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:13:54.742635 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.736953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:37039 (local address 127.16.99.254:38251)
0812 00:13:54.737040 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.737042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.737472 (+ 430us) server_negotiation.cc:407] Connection header received
0812 00:13:54.737803 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.737804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.737913 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.737994 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:54.739011 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.739705 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.740885 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.741104 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.741174 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.742250 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:54.742251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:54.742252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:54.742258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:54.742276 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:54.742280 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:54.742412 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.742524 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.742563 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:13:54.742784 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.736862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:38251 (local address 127.16.99.194:37039)
0812 00:13:54.737072 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.737119 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.737648 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.738219 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.738224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.738234 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:54.738899 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.738906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.739821 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.739822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.740778 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.740781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.741244 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.741245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.741379 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.741999 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:54.741999 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:54.742031 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:54.742058 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:54.742372 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:54.742377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:54.742381 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.742491 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.742587 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260812 00:13:54.743479 4471 heartbeater.cc:347] Connected to a master server at 127.16.99.254:38251
I20260812 00:13:54.743676 4471 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:54.743932 4471 heartbeater.cc:511] Master 127.16.99.254:38251 requested a full tablet report, sending...
I20260812 00:13:54.744333 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 4335
I20260812 00:13:54.744460 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:13:54.744575 4138 ts_manager.cc:194] Registered new tserver with Master: 1ef954d5ec3f4b61abb63c9bd8c72acf (127.16.99.194:44245)
I20260812 00:13:54.744692 16783 external_mini_cluster.cc:1428] Setting key 4af6926f3e7636b13cf9ce5d053efe0f
I20260812 00:13:54.745297 4138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.194:37039
I20260812 00:13:54.745857 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38251
--builtin_ntp_servers=127.16.99.212:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:54.853425 4474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:54.853717 4474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:54.853760 4474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:54.855216 4474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:54.855290 4474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:54.855338 4474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:54.856959 4474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:54.857283 4474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:54.857637 4474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:54.863830 4480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:54.863867 4474 server_base.cc:1034] running on GCE node
W20260812 00:13:54.863848 4479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:54.863863 4482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:54.864514 4474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:54.864861 4474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:54.866022 4474 hybrid_clock.cc:648] HybridClock initialized: now 1786493634865999 us; error 46 us; skew 500 ppm
I20260812 00:13:54.867763 4474 webserver.cc:460] Webserver started at http://127.16.99.195:43531/ using document root <none> and password file <none>
I20260812 00:13:54.868093 4474 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:54.868149 4474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:54.868394 4474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:54.869752 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "2d9d8e318f240a00cfa3a505df20d5f9"
server_key_iv: "fc61c23d1e89239dabfadb647fe5b69a"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.870182 4474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "2d9d8e318f240a00cfa3a505df20d5f9"
server_key_iv: "fc61c23d1e89239dabfadb647fe5b69a"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.872008 4474 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260812 00:13:54.873179 4488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.873463 4474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:54.873574 4474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474"
format_stamp: "Formatted at 2026-08-12 00:13:54 on dist-test-slave-zpfg"
server_key: "2d9d8e318f240a00cfa3a505df20d5f9"
server_key_iv: "fc61c23d1e89239dabfadb647fe5b69a"
server_key_version: "encryptionkey@0"
I20260812 00:13:54.873667 4474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:54.886715 4474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:54.887097 4474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:54.887259 4474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:54.887558 4474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:54.888077 4474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:54.888120 4474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.888163 4474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:54.888185 4474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:54.895231 4474 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:39785
I20260812 00:13:54.895346 4610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:39785 every 8 connection(s)
I20260812 00:13:54.895862 4474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:13:54.902532 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 4474
I20260812 00:13:54.902683 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:13:54.902905 16783 external_mini_cluster.cc:1428] Setting key 07b7a41ba50e202ae5898f2ff50affd3
I20260812 00:13:54.903664 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.896912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:45681 (local address 127.16.99.254:38251)
0812 00:13:54.897014 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.897016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.897317 (+ 301us) server_negotiation.cc:407] Connection header received
0812 00:13:54.898120 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.898121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.898134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.898192 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:54.899162 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.899910 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.901614 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.901902 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.902001 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.903279 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:54.903280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:54.903281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:54.903288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:54.903311 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:54.903317 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:54.903402 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.903529 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.903568 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 00:13:54.904033 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.896866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:38251 (local address 127.16.99.195:45681)
0812 00:13:54.897122 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.897172 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.897968 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.898297 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.898301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.898311 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:54.899008 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.899015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.900056 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.900058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.901497 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.901501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.902015 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.902016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.902183 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.902977 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:54.902977 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:54.903026 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:54.903084 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:54.903413 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:54.903421 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:54.903426 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.903538 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.903696 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":131,"threads_started":1}
I20260812 00:13:54.904563 4611 heartbeater.cc:347] Connected to a master server at 127.16.99.254:38251
I20260812 00:13:54.904696 4611 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:54.904889 4611 heartbeater.cc:511] Master 127.16.99.254:38251 requested a full tablet report, sending...
I20260812 00:13:54.905280 4138 ts_manager.cc:194] Registered new tserver with Master: 2d51df5a6b0b490ba7bdc0e6e2d98474 (127.16.99.195:39785)
I20260812 00:13:54.905663 4138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.195:45681
I20260812 00:13:54.915582 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:54.920945 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.917384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.16.99.254:38251)
0812 00:13:54.917446 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.917448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.917545 (+ 97us) server_negotiation.cc:407] Connection header received
0812 00:13:54.917606 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.917607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.917616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.917666 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:54.917870 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.918467 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.919435 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.919652 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.919737 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.920563 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:54.920564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:54.920565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:54.920571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:54.920591 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:54.920601 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:54.920714 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.920822 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.920843 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:13:54.923318 4138 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:54.925349 4138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:54.933308 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.927905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:39785 (local address 127.0.0.1:51960)
0812 00:13:54.928095 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.928136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.928195 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.928587 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.928591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.928594 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:54.928689 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.928692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.930390 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.930392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.931837 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.931840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.932868 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.932868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.932972 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.932988 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.933064 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.933123 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":154,"thread_start_us":68,"threads_started":1}
I20260812 00:13:54.933740 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.928227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.16.99.195:39785)
0812 00:13:54.928396 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.928401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.928413 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:13:54.928456 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.928458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.928475 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.928563 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:54.928799 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.930234 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.931939 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.932777 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.932854 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.932977 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.933545 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.933592 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260812 00:13:54.934259 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.927905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44245 (local address 127.0.0.1:48940)
0812 00:13:54.928095 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.928134 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.928194 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.930402 (+ 2208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.930403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.930406 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:54.930456 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.930458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.931606 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.931607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.932748 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.932752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.934003 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.934004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.934071 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.934074 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.934130 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.934183 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"spinlock_wait_cycles":768,"thread_start_us":67,"threads_started":1}
I20260812 00:13:54.934271 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.928640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.16.99.194:44245)
0812 00:13:54.928901 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.928907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.928917 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:13:54.928955 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.928957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.928973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.929059 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:54.930544 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.931508 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.932864 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.933774 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.933852 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.933964 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.934149 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.934189 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
I20260812 00:13:54.934413 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.927916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:42177 (local address 127.0.0.1:48178)
0812 00:13:54.928287 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:54.928295 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:54.928323 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:54.928937 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:54.928938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:54.928940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:54.928990 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.928992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.930390 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.930392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.931839 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:54.931840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.934115 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.934116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:54.934184 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.934189 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:54.934256 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 00:13:54.934304 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":29,"thread_start_us":61,"threads_started":1}
I20260812 00:13:54.934698 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:54.928437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.16.99.193:42177)
0812 00:13:54.928765 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:54.928771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:54.928783 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:13:54.928833 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:54.928835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:54.928855 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:54.928940 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:54.929069 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.930265 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.933026 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:54.933897 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:54.933973 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:54.934071 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:54.934548 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 00:13:54.934601 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
I20260812 00:13:54.934617 4541 tablet_service.cc:1467] Processing CreateTablet for tablet 83385d7732124663a2047cb6991bcb07 (DEFAULT_TABLE table=test-table [id=1a8dc246f1b547bfadc0fd3069bbf4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:54.935091 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83385d7732124663a2047cb6991bcb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:54.935102 4401 tablet_service.cc:1467] Processing CreateTablet for tablet 83385d7732124663a2047cb6991bcb07 (DEFAULT_TABLE table=test-table [id=1a8dc246f1b547bfadc0fd3069bbf4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:54.935477 4401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83385d7732124663a2047cb6991bcb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:54.935420 4264 tablet_service.cc:1467] Processing CreateTablet for tablet 83385d7732124663a2047cb6991bcb07 (DEFAULT_TABLE table=test-table [id=1a8dc246f1b547bfadc0fd3069bbf4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:54.935760 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83385d7732124663a2047cb6991bcb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:54.938323 4625 tablet_bootstrap.cc:492] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: Bootstrap starting.
I20260812 00:13:54.938948 4625 tablet_bootstrap.cc:654] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:54.939275 4626 tablet_bootstrap.cc:492] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Bootstrap starting.
I20260812 00:13:54.939308 4625 log.cc:824] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:54.939805 4627 tablet_bootstrap.cc:492] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031: Bootstrap starting.
I20260812 00:13:54.939980 4626 tablet_bootstrap.cc:654] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:54.940363 4626 log.cc:824] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:54.940529 4627 tablet_bootstrap.cc:654] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:54.940778 4625 tablet_bootstrap.cc:492] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: No bootstrap required, opened a new log
I20260812 00:13:54.940868 4625 ts_tablet_manager.cc:1397] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:13:54.940904 4627 log.cc:824] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:54.941759 4627 tablet_bootstrap.cc:492] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031: No bootstrap required, opened a new log
I20260812 00:13:54.941792 4626 tablet_bootstrap.cc:492] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: No bootstrap required, opened a new log
I20260812 00:13:54.941836 4627 ts_tablet_manager.cc:1397] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:13:54.941843 4626 ts_tablet_manager.cc:1397] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 00:13:54.943337 4625 raft_consensus.cc:348] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:54.943536 4625 raft_consensus.cc:374] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:54.943588 4625 raft_consensus.cc:729] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef954d5ec3f4b61abb63c9bd8c72acf, State: Initialized, Role: FOLLOWER
I20260812 00:13:54.943778 4625 consensus_queue.cc:260] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:54.944113 4625 ts_tablet_manager.cc:1428] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:13:54.944135 4471 heartbeater.cc:503] Master 127.16.99.254:38251 was elected leader, sending a full tablet report...
I20260812 00:13:54.944638 4627 raft_consensus.cc:348] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:54.944650 4626 raft_consensus.cc:348] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:54.944820 4626 raft_consensus.cc:374] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:54.944820 4627 raft_consensus.cc:374] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:54.944857 4626 raft_consensus.cc:729] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d51df5a6b0b490ba7bdc0e6e2d98474, State: Initialized, Role: FOLLOWER
I20260812 00:13:54.944868 4627 raft_consensus.cc:729] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6e110431fa4e7b81642724729d1031, State: Initialized, Role: FOLLOWER
I20260812 00:13:54.944991 4626 consensus_queue.cc:260] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:54.944991 4627 consensus_queue.cc:260] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:54.945286 4627 ts_tablet_manager.cc:1428] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:13:54.945286 4626 ts_tablet_manager.cc:1428] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:13:54.945327 4332 heartbeater.cc:503] Master 127.16.99.254:38251 was elected leader, sending a full tablet report...
I20260812 00:13:54.945410 4611 heartbeater.cc:503] Master 127.16.99.254:38251 was elected leader, sending a full tablet report...
W20260812 00:13:54.986801 4472 tablet.cc:2369] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:55.033720 4634 raft_consensus.cc:482] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:55.033864 4634 raft_consensus.cc:504] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:55.034227 4634 leader_election.cc:288] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6e110431fa4e7b81642724729d1031 (127.16.99.193:42177), 1ef954d5ec3f4b61abb63c9bd8c72acf (127.16.99.194:44245)
I20260812 00:13:55.037901 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:55.034626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44245 (local address 127.16.99.195:54939)
0812 00:13:55.034837 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:55.034846 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:55.034899 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:55.035153 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:55.035154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:55.035156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:55.035243 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:55.035245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.036014 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.036015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:55.036884 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:55.036887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.037525 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.037527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:55.037680 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:55.037685 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:55.037736 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 00:13:55.037771 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":148,"threads_started":1}
I20260812 00:13:55.037974 4624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:55.034736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:54939 (local address 127.16.99.194:44245)
0812 00:13:55.034790 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:55.034792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:55.034962 (+ 170us) server_negotiation.cc:407] Connection header received
0812 00:13:55.034998 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:55.034999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:55.035015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:55.035085 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:55.035335 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.035901 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.036991 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.037418 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.037485 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:55.037492 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:55.037801 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 00:13:55.037855 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:13:55.037981 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:55.034699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:43079 (local address 127.16.99.193:42177)
0812 00:13:55.034765 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:55.034767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:55.034775 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:13:55.034807 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:55.034808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:55.034818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:55.034887 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:55.035220 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.035779 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.036913 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.037522 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.037606 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:55.037616 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:55.037845 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 00:13:55.037893 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:13:55.038040 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:55.034561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:42177 (local address 127.16.99.195:43079)
0812 00:13:55.034637 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:55.034650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:55.034706 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:55.034961 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:55.034963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:55.034965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:55.035056 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:55.035058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.035866 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.035867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:55.036796 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:55.036799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.037628 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.037628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:55.037701 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:55.037705 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:55.037824 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 00:13:55.037871 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 00:13:55.038458 4422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83385d7732124663a2047cb6991bcb07" candidate_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" is_pre_election: true
I20260812 00:13:55.038458 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83385d7732124663a2047cb6991bcb07" candidate_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6e110431fa4e7b81642724729d1031" is_pre_election: true
I20260812 00:13:55.038642 4284 raft_consensus.cc:2454] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d51df5a6b0b490ba7bdc0e6e2d98474 in term 0.
I20260812 00:13:55.038642 4422 raft_consensus.cc:2454] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d51df5a6b0b490ba7bdc0e6e2d98474 in term 0.
I20260812 00:13:55.038898 4491 leader_election.cc:302] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef954d5ec3f4b61abb63c9bd8c72acf, 2d51df5a6b0b490ba7bdc0e6e2d98474; no voters:
I20260812 00:13:55.039038 4634 raft_consensus.cc:2781] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:55.039103 4634 raft_consensus.cc:482] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:55.039132 4634 raft_consensus.cc:3037] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:55.040038 4634 raft_consensus.cc:504] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:55.040194 4634 leader_election.cc:288] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [CANDIDATE]: Term 1 election: Requested vote from peers 4c6e110431fa4e7b81642724729d1031 (127.16.99.193:42177), 1ef954d5ec3f4b61abb63c9bd8c72acf (127.16.99.194:44245)
I20260812 00:13:55.040457 4284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83385d7732124663a2047cb6991bcb07" candidate_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6e110431fa4e7b81642724729d1031"
I20260812 00:13:55.040469 4422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83385d7732124663a2047cb6991bcb07" candidate_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf"
I20260812 00:13:55.040534 4284 raft_consensus.cc:3037] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:55.040591 4422 raft_consensus.cc:3037] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:55.041391 4284 raft_consensus.cc:2454] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d51df5a6b0b490ba7bdc0e6e2d98474 in term 1.
I20260812 00:13:55.041500 4422 raft_consensus.cc:2454] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d51df5a6b0b490ba7bdc0e6e2d98474 in term 1.
I20260812 00:13:55.041602 4489 leader_election.cc:302] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d51df5a6b0b490ba7bdc0e6e2d98474, 4c6e110431fa4e7b81642724729d1031; no voters:
I20260812 00:13:55.041752 4634 raft_consensus.cc:2781] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:55.041949 4634 raft_consensus.cc:686] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 1 LEADER]: Becoming Leader. State: Replica: 2d51df5a6b0b490ba7bdc0e6e2d98474, State: Running, Role: LEADER
I20260812 00:13:55.042089 4634 consensus_queue.cc:237] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } }
I20260812 00:13:55.043280 4136 catalog_manager.cc:5495] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d51df5a6b0b490ba7bdc0e6e2d98474 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d51df5a6b0b490ba7bdc0e6e2d98474" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 39785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:55.053545 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:55.050320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.16.99.195:39785)
0812 00:13:55.050528 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:55.050530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:55.050538 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:13:55.050580 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:55.050581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:55.050600 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:55.050711 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:55.050974 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.051543 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.052526 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:55.052780 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:55.052845 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:55.053004 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:55.053122 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:55.053315 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:55.053385 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 00:13:55.053397 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260812 00:13:55.055892 4422 raft_consensus.cc:1261] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 1 FOLLOWER]: Refusing update from remote peer 2d51df5a6b0b490ba7bdc0e6e2d98474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:55.055925 4284 raft_consensus.cc:1261] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 1 FOLLOWER]: Refusing update from remote peer 2d51df5a6b0b490ba7bdc0e6e2d98474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:55.056318 4634 consensus_queue.cc:1035] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6e110431fa4e7b81642724729d1031" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 42177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:55.056471 4636 consensus_queue.cc:1035] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ef954d5ec3f4b61abb63c9bd8c72acf" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:55.059320 4639 mvcc.cc:205] Tried to move back new op lower bound from 7317477929185361920 to 7317477929133252608. Current Snapshot: MvccSnapshot[applied={T|T < 7317477929185361920}]
I20260812 00:13:55.062502 4138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:55.062635 4138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:55.063741 4138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17b8915989724011afc81da8b86cf4e5: Sending DeleteTablet for 3 replicas of tablet 83385d7732124663a2047cb6991bcb07
I20260812 00:13:55.064131 4541 tablet_service.cc:1514] Processing DeleteTablet for tablet 83385d7732124663a2047cb6991bcb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:55 UTC) from {username='slave'} at 127.0.0.1:51960
I20260812 00:13:55.064257 4401 tablet_service.cc:1514] Processing DeleteTablet for tablet 83385d7732124663a2047cb6991bcb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:55 UTC) from {username='slave'} at 127.0.0.1:48940
I20260812 00:13:55.064296 4264 tablet_service.cc:1514] Processing DeleteTablet for tablet 83385d7732124663a2047cb6991bcb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:55 UTC) from {username='slave'} at 127.0.0.1:48178
I20260812 00:13:55.064538 4649 tablet_replica.cc:330] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: stopping tablet replica
I20260812 00:13:55.064594 4651 tablet_replica.cc:330] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031: stopping tablet replica
I20260812 00:13:55.064723 4651 raft_consensus.cc:2229] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:55.064757 4649 raft_consensus.cc:2229] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:13:55.064847 4651 raft_consensus.cc:2258] T 83385d7732124663a2047cb6991bcb07 P 4c6e110431fa4e7b81642724729d1031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:55.064898 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 4200
I20260812 00:13:55.065054 4649 raft_consensus.cc:2258] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:55.068779 4649 ts_tablet_manager.cc:1905] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:55.070480 4649 ts_tablet_manager.cc:1918] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:55.070587 4649 log.cc:1196] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/83385d7732124663a2047cb6991bcb07
I20260812 00:13:55.070753 4650 tablet_replica.cc:330] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: stopping tablet replica
I20260812 00:13:55.070914 4649 ts_tablet_manager.cc:1939] T 83385d7732124663a2047cb6991bcb07 P 2d51df5a6b0b490ba7bdc0e6e2d98474: Deleting consensus metadata
I20260812 00:13:55.071270 4650 raft_consensus.cc:2229] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:55.071424 4122 catalog_manager.cc:4841] TS 2d51df5a6b0b490ba7bdc0e6e2d98474 (127.16.99.195:39785): tablet 83385d7732124663a2047cb6991bcb07 (table test-table [id=1a8dc246f1b547bfadc0fd3069bbf4da]) successfully deleted
I20260812 00:13:55.071492 4650 raft_consensus.cc:2258] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:55.072089 4650 ts_tablet_manager.cc:1905] T 83385d7732124663a2047cb6991bcb07 P 1ef954d5ec3f4b61abb63c9bd8c72acf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 00:13:55.072774 4119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:42177 (error 108)
I20260812 00:13:55.072917 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 4335
I20260812 00:13:55.078454 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:55.078219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:42177 (local address 127.0.0.1:48182)
0812 00:13:55.078306 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:55.078406 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:42177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 00:13:55.078612 4119 catalog_manager.cc:4584] TS 4c6e110431fa4e7b81642724729d1031 (127.16.99.193:42177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83385d7732124663a2047cb6991bcb07: Network error: Client connection negotiation failed: client connection to 127.16.99.193:42177: connect: Connection refused (error 111)
I20260812 00:13:55.082942 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 4474
I20260812 00:13:55.083006 4620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:55.082851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44245 (local address 127.0.0.1:48952)
0812 00:13:55.082934 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:55.082972 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.194:44245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 00:13:55.088945 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 4104
2026-08-12T00:13:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (833 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:13:55 dist-test-slave-zpfg krb5kdc[4655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:13:55 dist-test-slave-zpfg krb5kdc[4655](info): set up 2 sockets
Aug 12 00:13:55 dist-test-slave-zpfg krb5kdc[4655](info): commencing operation
krb5kdc: starting...
W20260812 00:13:56.202394 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:13:56 dist-test-slave-zpfg krb5kdc[4655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:13:56.338429 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33061
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:56.424695 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:56.424995 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:56.425042 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:56.426390 4668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:56.426478 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:56.426502 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:56.426524 4668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:56.426553 4668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:56.428246 4668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:33061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:56.428587 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:56.428853 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:56.434443 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:56.434442 4677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:56.434474 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:56.434707 4668 server_base.cc:1034] running on GCE node
I20260812 00:13:56.434976 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:56.435366 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:56.436554 4668 hybrid_clock.cc:648] HybridClock initialized: now 1786493636436584 us; error 114 us; skew 500 ppm
Aug 12 00:13:56 dist-test-slave-zpfg krb5kdc[4655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:56.439780 4668 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:13:56.440289 4668 webserver.cc:460] Webserver started at http://127.16.99.254:46069/ using document root <none> and password file <none>
I20260812 00:13:56.440579 4668 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:56.440656 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:56.440828 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:56.442101 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "3da42c1d28d34f6cb4d892dbac72a1e3"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "0a13f7cb2ea4973b846ae74f2191185b"
server_key_iv: "4a951e4d50ee705ffe8346df8a8b4aed"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.442525 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "3da42c1d28d34f6cb4d892dbac72a1e3"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "0a13f7cb2ea4973b846ae74f2191185b"
server_key_iv: "4a951e4d50ee705ffe8346df8a8b4aed"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.444257 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:56.445328 4684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:56.445747 4668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:56.445885 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "3da42c1d28d34f6cb4d892dbac72a1e3"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "0a13f7cb2ea4973b846ae74f2191185b"
server_key_iv: "4a951e4d50ee705ffe8346df8a8b4aed"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.445986 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:56.455842 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:56.458539 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:56.458760 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:56.463270 4668 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:33061
I20260812 00:13:56.463335 4738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:33061 every 8 connection(s)
I20260812 00:13:56.463730 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:56.464736 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:56.464807 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 4668
I20260812 00:13:56.464941 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:13:56.465185 16783 external_mini_cluster.cc:1428] Setting key 2039dde1048ebd11ae40cd650bbb3271
I20260812 00:13:56.468032 4739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3: Bootstrap starting.
I20260812 00:13:56.468848 4739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:56.469240 4739 log.cc:824] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:56.470099 4739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3: No bootstrap required, opened a new log
I20260812 00:13:56.472806 4739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33061 } }
I20260812 00:13:56.473048 4739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:56.473079 4739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da42c1d28d34f6cb4d892dbac72a1e3, State: Initialized, Role: FOLLOWER
I20260812 00:13:56.473308 4739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33061 } }
I20260812 00:13:56.473424 4739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:56.473464 4739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:56.473510 4739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 00:13:56 dist-test-slave-zpfg krb5kdc[4655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:56.474323 4739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33061 } }
I20260812 00:13:56.474536 4739 leader_election.cc:302] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da42c1d28d34f6cb4d892dbac72a1e3; no voters:
I20260812 00:13:56.474794 4739 leader_election.cc:288] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:56.474843 4744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:56.474987 4744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [term 1 LEADER]: Becoming Leader. State: Replica: 3da42c1d28d34f6cb4d892dbac72a1e3, State: Running, Role: LEADER
I20260812 00:13:56.475102 4744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33061 } }
I20260812 00:13:56.475154 4739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:56.475714 4746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33061 } } }
I20260812 00:13:56.475809 4746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:56.475916 4745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da42c1d28d34f6cb4d892dbac72a1e3. Latest consensus state: current_term: 1 leader_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da42c1d28d34f6cb4d892dbac72a1e3" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 33061 } } }
I20260812 00:13:56.477128 4745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3 [sys.catalog]: This master's current role is: LEADER
W20260812 00:13:56.477576 4760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:13:56.477634 4760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:13:56.478955 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:56.465915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.16.99.254:33061)
0812 00:13:56.466143 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:56.466152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:56.466651 (+ 499us) server_negotiation.cc:407] Connection header received
0812 00:13:56.467267 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:56.467275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:56.467333 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:56.467509 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:56.468265 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.469795 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.470777 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.471466 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.471614 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:56.474515 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:56.474520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:56.474520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:56.474534 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:56.476985 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:56.477543 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:56.477546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:56.477546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:56.477663 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:56.477944 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:56.477945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:56.477946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:56.478276 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:56.478453 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:56.478585 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 00:13:56.478733 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260812 00:13:56.482525 4762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:56.482760 4762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:56.484587 4762 catalog_manager.cc:1347] Generated new cluster ID: 7c5c455103b84ab8bd348ce023185bd4
I20260812 00:13:56.484647 4762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:56.491420 4762 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:56.492170 4762 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:56.506059 4762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3: Generated new TSK 0
I20260812 00:13:56.506330 4762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.99.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
Entry for principal kudu/127.16.99.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab.
I20260812 00:13:56.541962 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33061
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:56.627949 4767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:56.628197 4767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:56.628268 4767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:56.629492 4767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:56.629563 4767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:56.629616 4767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:56.631250 4767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.193.keytab
--principal=kudu/127.16.99.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:56.631552 4767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:56.631803 4767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:56.637445 4772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:56.637455 4773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:56.637454 4775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:56.637888 4767 server_base.cc:1034] running on GCE node
I20260812 00:13:56.638144 4767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:56.638484 4767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:56.639658 4767 hybrid_clock.cc:648] HybridClock initialized: now 1786493636639635 us; error 40 us; skew 500 ppm
Aug 12 00:13:56 dist-test-slave-zpfg krb5kdc[4655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:56.642526 4767 init.cc:377] Logged in from keytab as kudu/127.16.99.193@KRBTEST.COM (short username kudu)
I20260812 00:13:56.643060 4767 webserver.cc:460] Webserver started at http://127.16.99.193:35675/ using document root <none> and password file <none>
I20260812 00:13:56.643299 4767 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:56.643379 4767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:56.643522 4767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:56.644702 4767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "f840cef836a24a21a583cf932aec253e"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "b6c8e77d2eba1a75afd981295c93df26"
server_key_iv: "148b9385e8224367dfc7501afd3b964b"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.645071 4767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "f840cef836a24a21a583cf932aec253e"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "b6c8e77d2eba1a75afd981295c93df26"
server_key_iv: "148b9385e8224367dfc7501afd3b964b"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.646524 4767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:13:56.647514 4783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:56.647769 4767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:56.647890 4767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "f840cef836a24a21a583cf932aec253e"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "b6c8e77d2eba1a75afd981295c93df26"
server_key_iv: "148b9385e8224367dfc7501afd3b964b"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.647991 4767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:56.661216 4767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:56.663460 4767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:56.663594 4767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:56.663817 4767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:56.664211 4767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:56.664258 4767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:56.664283 4767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:56.664322 4767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:56.671438 4767 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:37311
I20260812 00:13:56.671517 4900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:37311 every 8 connection(s)
I20260812 00:13:56.671906 4767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:56.678669 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 4767
I20260812 00:13:56.678779 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:13:56.679020 16783 external_mini_cluster.cc:1428] Setting key 9ce2cd570490305f85f3ab0376b9f50c
Aug 12 00:13:56 dist-test-slave-zpfg krb5kdc[4655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.193@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:56.683770 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:56.672747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:54079 (local address 127.16.99.254:33061)
0812 00:13:56.672861 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:56.672864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:56.673043 (+ 179us) server_negotiation.cc:407] Connection header received
0812 00:13:56.673682 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:56.673684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:56.673700 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:56.673769 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:56.675105 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.675831 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.677087 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.677447 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.677508 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:56.680280 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:56.680282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:56.680282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:56.680291 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:56.682015 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:56.682616 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:56.682617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:56.682617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:56.682718 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:56.683042 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:56.683043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:56.683044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:56.683160 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:56.683225 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:56.683554 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 00:13:56.683660 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 00:13:56.684051 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:56.672682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33061 (local address 127.16.99.193:54079)
0812 00:13:56.672870 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:56.672933 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:56.673517 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:56.673860 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:56.673864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:56.674767 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:56.675005 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:56.675011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.675928 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.675930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:56.676987 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:56.676990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.677720 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.677721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:56.677851 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:56.678475 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:56.678483 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:56.680180 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:56.682118 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:56.682127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:56.682129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:56.682534 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:56.682795 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:56.682796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:56.682796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:56.682966 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:56.683229 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:56.683235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:56.683396 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 00:13:56.683508 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 00:13:56.683751 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
I20260812 00:13:56.684664 4901 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33061
I20260812 00:13:56.684762 4901 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:56.684970 4901 heartbeater.cc:511] Master 127.16.99.254:33061 requested a full tablet report, sending...
I20260812 00:13:56.685536 4702 ts_manager.cc:194] Registered new tserver with Master: f840cef836a24a21a583cf932aec253e (127.16.99.193:37311)
I20260812 00:13:56.686398 4702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.193@KRBTEST.COM'} at 127.16.99.193:54079
WARNING: no policy specified for kudu/127.16.99.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
Entry for principal kudu/127.16.99.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab.
I20260812 00:13:56.710315 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33061
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:56.806263 4906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:56.806591 4906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:56.806631 4906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:56.807996 4906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:56.808043 4906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:56.808073 4906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:56.809881 4906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.194.keytab
--principal=kudu/127.16.99.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:56.810173 4906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:56.810489 4906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:56.816578 4914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:56.816586 4912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:56.816617 4906 server_base.cc:1034] running on GCE node
W20260812 00:13:56.816581 4911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:56.817255 4906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:56.817546 4906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:56.818698 4906 hybrid_clock.cc:648] HybridClock initialized: now 1786493636818666 us; error 37 us; skew 500 ppm
Aug 12 00:13:56 dist-test-slave-zpfg krb5kdc[4655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:56.821620 4906 init.cc:377] Logged in from keytab as kudu/127.16.99.194@KRBTEST.COM (short username kudu)
I20260812 00:13:56.822322 4906 webserver.cc:460] Webserver started at http://127.16.99.194:37609/ using document root <none> and password file <none>
I20260812 00:13:56.822671 4906 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:56.822754 4906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:56.822911 4906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:56.824182 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "4c6b86e5f2d04a34a426c21dda81e917"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "6492613afcd46b742a06fa7e89b8b1d3"
server_key_iv: "0dd258acd34225c83f2d74ce5748c2cc"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.824566 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "4c6b86e5f2d04a34a426c21dda81e917"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "6492613afcd46b742a06fa7e89b8b1d3"
server_key_iv: "0dd258acd34225c83f2d74ce5748c2cc"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.826135 4906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:13:56.827179 4921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:56.827360 4906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:13:56.827472 4906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "4c6b86e5f2d04a34a426c21dda81e917"
format_stamp: "Formatted at 2026-08-12 00:13:56 on dist-test-slave-zpfg"
server_key: "6492613afcd46b742a06fa7e89b8b1d3"
server_key_iv: "0dd258acd34225c83f2d74ce5748c2cc"
server_key_version: "encryptionkey@0"
I20260812 00:13:56.827553 4906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:56.859005 4906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:56.861032 4906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:56.861232 4906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:56.861507 4906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:56.861857 4906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:56.861928 4906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:56.861971 4906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:56.862000 4906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:56.868127 4906 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:44201
I20260812 00:13:56.868178 5038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:44201 every 8 connection(s)
I20260812 00:13:56.868511 4906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
Aug 12 00:13:56 dist-test-slave-zpfg krb5kdc[4655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493636, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.194@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:56.877779 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 4906
I20260812 00:13:56.877923 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:13:56.878389 16783 external_mini_cluster.cc:1428] Setting key 4eb84b10d6fe415e002cd054a3929bf9
I20260812 00:13:56.879082 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:56.869279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:35773 (local address 127.16.99.254:33061)
0812 00:13:56.869369 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:56.869372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:56.869625 (+ 253us) server_negotiation.cc:407] Connection header received
0812 00:13:56.870146 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:56.870147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:56.870158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:56.870215 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:56.871394 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.871913 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.873013 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.873201 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.873277 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:56.875614 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:56.875615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:56.875615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:56.875623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:56.877324 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:56.877811 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:56.877812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:56.877813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:56.877892 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:56.878223 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:56.878224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:56.878225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:56.878432 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:56.878520 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:56.878902 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 00:13:56.878993 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 00:13:56.879309 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:56.869249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33061 (local address 127.16.99.194:35773)
0812 00:13:56.869487 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:56.869528 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:56.870055 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:56.870431 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:56.870436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:56.871082 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:56.871300 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:56.871305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.872007 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.872007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:56.872918 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:56.872921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:56.873330 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:56.873331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:56.873441 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:56.873969 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:56.873978 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:56.875517 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:56.877414 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:56.877421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:56.877423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:56.877658 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:56.877964 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:56.877965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:56.877965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:56.878091 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:56.878527 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:56.878535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:56.878737 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 00:13:56.878862 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 00:13:56.879036 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
I20260812 00:13:56.879928 5039 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33061
I20260812 00:13:56.880043 5039 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:56.880390 5039 heartbeater.cc:511] Master 127.16.99.254:33061 requested a full tablet report, sending...
I20260812 00:13:56.880831 4702 ts_manager.cc:194] Registered new tserver with Master: 4c6b86e5f2d04a34a426c21dda81e917 (127.16.99.194:44201)
I20260812 00:13:56.881436 4702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.194@KRBTEST.COM'} at 127.16.99.194:35773
WARNING: no policy specified for kudu/127.16.99.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
Entry for principal kudu/127.16.99.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab.
I20260812 00:13:56.910276 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33061
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:56.997334 5044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:56.997571 5044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:56.997599 5044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:56.998775 5044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:56.998821 5044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:56.998849 5044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:57.000356 5044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.195.keytab
--principal=kudu/127.16.99.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:57.000567 5044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:57.000785 5044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:57.006875 5052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.006875 5051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.006887 5054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:57.007220 5044 server_base.cc:1034] running on GCE node
I20260812 00:13:57.007483 5044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:57.007761 5044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:57.008920 5044 hybrid_clock.cc:648] HybridClock initialized: now 1786493637008894 us; error 51 us; skew 500 ppm
Aug 12 00:13:57 dist-test-slave-zpfg krb5kdc[4655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:57.011914 5044 init.cc:377] Logged in from keytab as kudu/127.16.99.195@KRBTEST.COM (short username kudu)
I20260812 00:13:57.012627 5044 webserver.cc:460] Webserver started at http://127.16.99.195:43717/ using document root <none> and password file <none>
I20260812 00:13:57.012867 5044 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:57.012938 5044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:57.013100 5044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:57.014211 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "9ada3579870f4c40a3e8e3b00dff9ea6"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "9cf4b297857c5c428ed71e35349badf1"
server_key_iv: "8ff6e5c3221f16d0daf18020be811b43"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.014683 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "9ada3579870f4c40a3e8e3b00dff9ea6"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "9cf4b297857c5c428ed71e35349badf1"
server_key_iv: "8ff6e5c3221f16d0daf18020be811b43"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.016387 5044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:57.017416 5061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.017664 5044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:57.017779 5044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "9ada3579870f4c40a3e8e3b00dff9ea6"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "9cf4b297857c5c428ed71e35349badf1"
server_key_iv: "8ff6e5c3221f16d0daf18020be811b43"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.017859 5044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:57.028703 5044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:57.031400 5044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:57.031579 5044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:57.031857 5044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:57.032197 5044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:57.032258 5044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.032303 5044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:57.032337 5044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.038983 5044 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:46465
I20260812 00:13:57.039049 5177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:46465 every 8 connection(s)
I20260812 00:13:57.039407 5044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
Aug 12 00:13:57 dist-test-slave-zpfg krb5kdc[4655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493637, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.195@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:13:57.046993 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 5044
I20260812 00:13:57.047086 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:13:57.047328 16783 external_mini_cluster.cc:1428] Setting key b6de98bdaf567668a4fd341f1eb187db
I20260812 00:13:57.050462 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.040299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:54427 (local address 127.16.99.254:33061)
0812 00:13:57.040349 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.040351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.040580 (+ 229us) server_negotiation.cc:407] Connection header received
0812 00:13:57.041147 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.041150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.041162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.041239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:57.042282 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.042809 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.043838 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.044052 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.044138 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.046841 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:57.046842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:57.046842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:57.046850 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:57.048809 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:57.049284 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:57.049285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:57.049285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:57.049386 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:57.049816 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:57.049816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:57.049817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:57.049933 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:57.049980 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.050239 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.050324 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:13:57.050675 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.040267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:33061 (local address 127.16.99.195:54427)
0812 00:13:57.040442 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.040479 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.040964 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.041293 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.041297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.041889 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:57.042112 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.042117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.042900 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.042901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.043738 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.043740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.044221 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.044222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.044352 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.044884 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:13:57.044893 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:57.046714 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:57.048891 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:57.048899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:57.048901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:57.049203 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:57.049452 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:13:57.049453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:13:57.049453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:13:57.049715 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:13:57.049980 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:57.049983 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:57.050113 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.050206 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.050427 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260812 00:13:57.051196 5178 heartbeater.cc:347] Connected to a master server at 127.16.99.254:33061
I20260812 00:13:57.051301 5178 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:57.051517 5178 heartbeater.cc:511] Master 127.16.99.254:33061 requested a full tablet report, sending...
I20260812 00:13:57.051955 4702 ts_manager.cc:194] Registered new tserver with Master: 9ada3579870f4c40a3e8e3b00dff9ea6 (127.16.99.195:46465)
I20260812 00:13:57.052368 4702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.99.195@KRBTEST.COM'} at 127.16.99.195:54427
I20260812 00:13:57.059916 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:57.067800 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.061407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.16.99.254:33061)
0812 00:13:57.061486 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.061488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.061562 (+ 74us) server_negotiation.cc:407] Connection header received
0812 00:13:57.061606 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.061607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.061616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.061672 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:57.062231 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.062826 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.063542 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.063735 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.063793 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.065108 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:57.065109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:57.065110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:57.065121 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:57.066684 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:57.067035 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:57.067036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:57.067036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:57.067094 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:57.067257 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:57.067257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:57.067258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:57.067409 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:57.067484 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.067590 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.067662 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 00:13:57.069564 4702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:57.071358 4702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:57.078963 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.074571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.16.99.193:37311)
0812 00:13:57.074746 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.074751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.074762 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:13:57.074835 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.074838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.074855 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.074912 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.075192 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.076454 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.077774 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.078516 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.078602 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.078675 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.078824 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.078872 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260812 00:13:57.078971 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.074264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:37311 (local address 127.0.0.1:42324)
0812 00:13:57.074534 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.074579 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.074702 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.075040 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.075044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.075047 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.075120 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.075122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.076567 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.076568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.077685 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.077688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.078613 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.078614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.078704 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.078721 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.078798 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.078853 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":40,"thread_start_us":56,"threads_started":1}
I20260812 00:13:57.079192 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.074308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.16.99.195:46465)
0812 00:13:57.074534 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.074539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.074907 (+ 368us) server_negotiation.cc:407] Connection header received
0812 00:13:57.075022 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.075025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.075044 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.075125 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.075336 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.076233 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.077838 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.078549 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.078608 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.078666 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.079038 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.079084 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
I20260812 00:13:57.079769 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.074589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.16.99.194:44201)
0812 00:13:57.074865 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.074869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.075174 (+ 305us) server_negotiation.cc:407] Connection header received
0812 00:13:57.075221 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.075223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.075236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.075310 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.075455 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.076619 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.078034 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.078637 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.078717 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.078774 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.079013 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.079703 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
I20260812 00:13:57.078995 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.074291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:46465 (local address 127.0.0.1:53946)
0812 00:13:57.074825 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.074836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.074941 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.075214 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.075214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.075216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.075252 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.075253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.076342 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.076344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.077749 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.077752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.078704 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.078705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.078793 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.078798 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.078852 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.078903 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":59,"thread_start_us":140,"threads_started":1}
I20260812 00:13:57.079783 4833 tablet_service.cc:1467] Processing CreateTablet for tablet bf1cc9e866d0499c803d76763b3c9d47 (DEFAULT_TABLE table=test-table [id=302cd577dfd94fed831212306626b054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:57.080107 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.074318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44201 (local address 127.0.0.1:40984)
0812 00:13:57.075058 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.075068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.075112 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.075338 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.075339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.075341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.075376 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.075377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.076735 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.076736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.077945 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.077947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.078811 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.078812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.078863 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.078867 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.078913 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.078951 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":93,"thread_start_us":119,"threads_started":1}
I20260812 00:13:57.080200 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1cc9e866d0499c803d76763b3c9d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:57.082823 5110 tablet_service.cc:1467] Processing CreateTablet for tablet bf1cc9e866d0499c803d76763b3c9d47 (DEFAULT_TABLE table=test-table [id=302cd577dfd94fed831212306626b054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:57.083045 5196 tablet_bootstrap.cc:492] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: Bootstrap starting.
I20260812 00:13:57.083209 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1cc9e866d0499c803d76763b3c9d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:57.083874 5196 tablet_bootstrap.cc:654] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:57.084470 5196 log.cc:824] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:57.085359 5196 tablet_bootstrap.cc:492] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: No bootstrap required, opened a new log
I20260812 00:13:57.085431 5196 ts_tablet_manager.cc:1397] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:13:57.086934 5198 tablet_bootstrap.cc:492] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Bootstrap starting.
I20260812 00:13:57.087579 5196 raft_consensus.cc:348] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.087756 5196 raft_consensus.cc:374] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:57.087801 5196 raft_consensus.cc:729] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f840cef836a24a21a583cf932aec253e, State: Initialized, Role: FOLLOWER
I20260812 00:13:57.087852 5198 tablet_bootstrap.cc:654] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:57.087735 4970 tablet_service.cc:1467] Processing CreateTablet for tablet bf1cc9e866d0499c803d76763b3c9d47 (DEFAULT_TABLE table=test-table [id=302cd577dfd94fed831212306626b054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:57.087944 5196 consensus_queue.cc:260] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.088122 4970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1cc9e866d0499c803d76763b3c9d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:57.088287 5196 ts_tablet_manager.cc:1428] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:13:57.088323 4901 heartbeater.cc:503] Master 127.16.99.254:33061 was elected leader, sending a full tablet report...
I20260812 00:13:57.088533 5198 log.cc:824] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:57.089444 5198 tablet_bootstrap.cc:492] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: No bootstrap required, opened a new log
I20260812 00:13:57.089515 5198 ts_tablet_manager.cc:1397] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 00:13:57.091728 5201 tablet_bootstrap.cc:492] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Bootstrap starting.
I20260812 00:13:57.091697 5198 raft_consensus.cc:348] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.091910 5198 raft_consensus.cc:374] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:57.091976 5198 raft_consensus.cc:729] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ada3579870f4c40a3e8e3b00dff9ea6, State: Initialized, Role: FOLLOWER
I20260812 00:13:57.092108 5198 consensus_queue.cc:260] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.092437 5198 ts_tablet_manager.cc:1428] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 00:13:57.092465 5178 heartbeater.cc:503] Master 127.16.99.254:33061 was elected leader, sending a full tablet report...
I20260812 00:13:57.092481 5201 tablet_bootstrap.cc:654] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:57.093199 5201 log.cc:824] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:57.093931 5201 tablet_bootstrap.cc:492] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: No bootstrap required, opened a new log
I20260812 00:13:57.094013 5201 ts_tablet_manager.cc:1397] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 00:13:57.095650 5201 raft_consensus.cc:348] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.095819 5201 raft_consensus.cc:374] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:57.095871 5201 raft_consensus.cc:729] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6b86e5f2d04a34a426c21dda81e917, State: Initialized, Role: FOLLOWER
I20260812 00:13:57.095995 5201 consensus_queue.cc:260] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.096304 5201 ts_tablet_manager.cc:1428] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 00:13:57.096346 5039 heartbeater.cc:503] Master 127.16.99.254:33061 was elected leader, sending a full tablet report...
W20260812 00:13:57.119269 5040 tablet.cc:2369] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 00:13:57.172803 4902 tablet.cc:2369] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 00:13:57.221812 5202 raft_consensus.cc:482] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:57.221992 5202 raft_consensus.cc:504] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.222502 5202 leader_election.cc:288] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c6b86e5f2d04a34a426c21dda81e917 (127.16.99.194:44201), f840cef836a24a21a583cf932aec253e (127.16.99.193:37311)
I20260812 00:13:57.225934 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.222869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:41889 (local address 127.16.99.193:37311)
0812 00:13:57.222917 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.222919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.223151 (+ 232us) server_negotiation.cc:407] Connection header received
0812 00:13:57.223252 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.223253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.223268 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.223340 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.223512 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.223977 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.225060 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.225559 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.225648 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.225660 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.225797 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.225846 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:13:57.225950 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.222768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:37311 (local address 127.16.99.195:41889)
0812 00:13:57.223051 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.223062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.223125 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.223362 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.223363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.223365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.223442 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.223444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.224083 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.224084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.224948 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.224951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.225637 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.225639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.225724 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.225729 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.225786 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.225834 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":59,"thread_start_us":148,"threads_started":1}
I20260812 00:13:57.225970 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.222755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:44201 (local address 127.16.99.195:43161)
0812 00:13:57.222816 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.222829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.222888 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.223145 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.223147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.223149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.223264 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.223267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.224049 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.224050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.225050 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.225053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.225738 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.225739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.225823 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.225825 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.225873 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.225904 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:13:57.226047 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.222844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:43161 (local address 127.16.99.194:44201)
0812 00:13:57.222894 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.222896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.222950 (+ 54us) server_negotiation.cc:407] Connection header received
0812 00:13:57.222989 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.222990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.223003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.223066 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.223372 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.223957 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.225154 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.225660 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.225729 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.225738 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.225916 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.225963 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:13:57.226395 4990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1cc9e866d0499c803d76763b3c9d47" candidate_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6b86e5f2d04a34a426c21dda81e917" is_pre_election: true
I20260812 00:13:57.226585 4990 raft_consensus.cc:2454] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ada3579870f4c40a3e8e3b00dff9ea6 in term 0.
I20260812 00:13:57.226572 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1cc9e866d0499c803d76763b3c9d47" candidate_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f840cef836a24a21a583cf932aec253e" is_pre_election: true
I20260812 00:13:57.226770 4853 raft_consensus.cc:2454] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ada3579870f4c40a3e8e3b00dff9ea6 in term 0.
I20260812 00:13:57.226822 5065 leader_election.cc:302] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c6b86e5f2d04a34a426c21dda81e917, 9ada3579870f4c40a3e8e3b00dff9ea6; no voters:
I20260812 00:13:57.226984 5202 raft_consensus.cc:2781] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:57.227038 5202 raft_consensus.cc:482] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:57.227054 5202 raft_consensus.cc:3037] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:57.227824 5202 raft_consensus.cc:504] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.227983 5202 leader_election.cc:288] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 4c6b86e5f2d04a34a426c21dda81e917 (127.16.99.194:44201), f840cef836a24a21a583cf932aec253e (127.16.99.193:37311)
I20260812 00:13:57.228168 4990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1cc9e866d0499c803d76763b3c9d47" candidate_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c6b86e5f2d04a34a426c21dda81e917"
I20260812 00:13:57.228200 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1cc9e866d0499c803d76763b3c9d47" candidate_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f840cef836a24a21a583cf932aec253e"
I20260812 00:13:57.228298 4853 raft_consensus.cc:3037] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:57.228276 4990 raft_consensus.cc:3037] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:57.229059 4853 raft_consensus.cc:2454] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ada3579870f4c40a3e8e3b00dff9ea6 in term 1.
I20260812 00:13:57.229135 4990 raft_consensus.cc:2454] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ada3579870f4c40a3e8e3b00dff9ea6 in term 1.
I20260812 00:13:57.229249 5063 leader_election.cc:302] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ada3579870f4c40a3e8e3b00dff9ea6, f840cef836a24a21a583cf932aec253e; no voters:
I20260812 00:13:57.229336 5202 raft_consensus.cc:2781] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:57.229496 5202 raft_consensus.cc:686] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ada3579870f4c40a3e8e3b00dff9ea6, State: Running, Role: LEADER
I20260812 00:13:57.229614 5202 consensus_queue.cc:237] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } }
I20260812 00:13:57.230644 4702 catalog_manager.cc:5495] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ada3579870f4c40a3e8e3b00dff9ea6 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ada3579870f4c40a3e8e3b00dff9ea6" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:57.267038 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.264411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.16.99.195:46465)
0812 00:13:57.264492 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.264494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.264502 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:13:57.264540 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.264541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.264561 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.264658 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:57.264820 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.265329 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.266120 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.266324 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.266465 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.266651 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:57.266765 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:57.266866 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.266886 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.266910 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 00:13:57.269413 4853 raft_consensus.cc:1261] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 1 FOLLOWER]: Refusing update from remote peer 9ada3579870f4c40a3e8e3b00dff9ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:57.269505 4990 raft_consensus.cc:1261] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 1 FOLLOWER]: Refusing update from remote peer 9ada3579870f4c40a3e8e3b00dff9ea6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:57.269851 5202 consensus_queue.cc:1035] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f840cef836a24a21a583cf932aec253e" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:57.270035 5206 consensus_queue.cc:1035] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c6b86e5f2d04a34a426c21dda81e917" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:57.273942 5211 mvcc.cc:205] Tried to move back new op lower bound from 7317477938251739136 to 7317477938093457408. Current Snapshot: MvccSnapshot[applied={T|T < 7317477938251739136 or (T in {7317477938251739136})}]
I20260812 00:13:57.275321 4701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:57.275434 4701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:57.276384 4701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3da42c1d28d34f6cb4d892dbac72a1e3: Sending DeleteTablet for 3 replicas of tablet bf1cc9e866d0499c803d76763b3c9d47
I20260812 00:13:57.276784 5110 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1cc9e866d0499c803d76763b3c9d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:57 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:53946
I20260812 00:13:57.276854 4833 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1cc9e866d0499c803d76763b3c9d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:57 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:42324
I20260812 00:13:57.276886 4970 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1cc9e866d0499c803d76763b3c9d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:57 UTC) from {username='kudu', principal='kudu/127.16.99.254@KRBTEST.COM'} at 127.0.0.1:40984
I20260812 00:13:57.277057 5219 tablet_replica.cc:330] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: stopping tablet replica
I20260812 00:13:57.277179 5219 raft_consensus.cc:2229] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:57.277338 5219 raft_consensus.cc:2258] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:57.277539 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 4767
I20260812 00:13:57.277806 5219 ts_tablet_manager.cc:1905] T bf1cc9e866d0499c803d76763b3c9d47 P f840cef836a24a21a583cf932aec253e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:57.277560 5220 tablet_replica.cc:330] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: stopping tablet replica
I20260812 00:13:57.278242 5220 raft_consensus.cc:2229] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:57.278316 5220 raft_consensus.cc:2258] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:57.280053 5218 tablet_replica.cc:330] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: stopping tablet replica
I20260812 00:13:57.280156 5220 ts_tablet_manager.cc:1905] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:57.280229 5218 raft_consensus.cc:2229] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:13:57.280386 5218 raft_consensus.cc:2258] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:57.280897 5218 ts_tablet_manager.cc:1905] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:57.282167 5220 ts_tablet_manager.cc:1918] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:57.282313 5220 log.cc:1196] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/bf1cc9e866d0499c803d76763b3c9d47
I20260812 00:13:57.282766 5220 ts_tablet_manager.cc:1939] T bf1cc9e866d0499c803d76763b3c9d47 P 4c6b86e5f2d04a34a426c21dda81e917: Deleting consensus metadata
I20260812 00:13:57.282838 5218 ts_tablet_manager.cc:1918] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:57.282939 5218 log.cc:1196] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/bf1cc9e866d0499c803d76763b3c9d47
I20260812 00:13:57.283310 5218 ts_tablet_manager.cc:1939] T bf1cc9e866d0499c803d76763b3c9d47 P 9ada3579870f4c40a3e8e3b00dff9ea6: Deleting consensus metadata
I20260812 00:13:57.283780 4685 catalog_manager.cc:4841] TS 9ada3579870f4c40a3e8e3b00dff9ea6 (127.16.99.195:46465): tablet bf1cc9e866d0499c803d76763b3c9d47 (table test-table [id=302cd577dfd94fed831212306626b054]) successfully deleted
I20260812 00:13:57.286530 4688 catalog_manager.cc:4841] TS 4c6b86e5f2d04a34a426c21dda81e917 (127.16.99.194:44201): tablet bf1cc9e866d0499c803d76763b3c9d47 (table test-table [id=302cd577dfd94fed831212306626b054]) successfully deleted
W20260812 00:13:57.287592 4686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:37311 (error 108)
I20260812 00:13:57.287981 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 4906
I20260812 00:13:57.288312 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.288074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:37311 (local address 127.0.0.1:47464)
0812 00:13:57.288132 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.288271 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 00:13:57.288533 4686 catalog_manager.cc:4584] TS f840cef836a24a21a583cf932aec253e (127.16.99.193:37311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf1cc9e866d0499c803d76763b3c9d47: Network error: Client connection negotiation failed: client connection to 127.16.99.193:37311: connect: Connection refused (error 111)
I20260812 00:13:57.299141 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 5044
I20260812 00:13:57.306710 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 4668
2026-08-12T00:13:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2216 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T00:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:57Z Disabled control of system clock
I20260812 00:13:57.337692 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37367
--webserver_interface=127.16.99.254
--webserver_port=0
--builtin_ntp_servers=127.16.99.212:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:57.421331 5225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:57.421581 5225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:57.421602 5225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:57.422847 5225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:57.422904 5225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:57.422917 5225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:57.422930 5225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:57.422942 5225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:57.424383 5225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:37367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:57.424623 5225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:57.424882 5225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:57.430950 5230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.430979 5233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.430976 5231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:57.431234 5225 server_base.cc:1034] running on GCE node
I20260812 00:13:57.431571 5225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:57.431936 5225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:57.433218 5225 hybrid_clock.cc:648] HybridClock initialized: now 1786493637433096 us; error 59 us; skew 500 ppm
I20260812 00:13:57.435051 5225 webserver.cc:460] Webserver started at http://127.16.99.254:41835/ using document root <none> and password file <none>
I20260812 00:13:57.435480 5225 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:57.435575 5225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:57.435760 5225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:57.437021 5225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "fc1304a851a0412d9bbb6c91d1c7552d"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "10de235d26ab6362d82386068c350b74"
server_key_iv: "8651bf34fc7a1ca3ab9a4c472b5db219"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.437438 5225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "fc1304a851a0412d9bbb6c91d1c7552d"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "10de235d26ab6362d82386068c350b74"
server_key_iv: "8651bf34fc7a1ca3ab9a4c472b5db219"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.439209 5225 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260812 00:13:57.440131 5239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.440377 5225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:57.440505 5225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "fc1304a851a0412d9bbb6c91d1c7552d"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "10de235d26ab6362d82386068c350b74"
server_key_iv: "8651bf34fc7a1ca3ab9a4c472b5db219"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.440610 5225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:57.456929 5225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:57.457345 5225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:57.457535 5225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:57.462159 5225 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:37367
I20260812 00:13:57.462214 5292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:37367 every 8 connection(s)
I20260812 00:13:57.462651 5225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:57.463733 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:57.466073 5293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d: Bootstrap starting.
I20260812 00:13:57.466739 5293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:57.467005 5293 log.cc:824] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:57.467850 5293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d: No bootstrap required, opened a new log
I20260812 00:13:57.469537 5293 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37367 } }
I20260812 00:13:57.469686 5293 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:57.469708 5293 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1304a851a0412d9bbb6c91d1c7552d, State: Initialized, Role: FOLLOWER
I20260812 00:13:57.469879 5293 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37367 } }
I20260812 00:13:57.469969 5293 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:57.469995 5293 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:57.470033 5293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:57.470623 5293 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37367 } }
I20260812 00:13:57.470726 5293 leader_election.cc:302] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc1304a851a0412d9bbb6c91d1c7552d; no voters:
I20260812 00:13:57.470902 5293 leader_election.cc:288] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:57.470945 5296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:57.471077 5296 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [term 1 LEADER]: Becoming Leader. State: Replica: fc1304a851a0412d9bbb6c91d1c7552d, State: Running, Role: LEADER
I20260812 00:13:57.471191 5293 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:57.471316 5296 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37367 } }
I20260812 00:13:57.471709 5296 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc1304a851a0412d9bbb6c91d1c7552d. Latest consensus state: current_term: 1 leader_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37367 } } }
I20260812 00:13:57.471815 5298 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1304a851a0412d9bbb6c91d1c7552d" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37367 } } }
I20260812 00:13:57.471803 5296 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:57.471925 5298 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:57.472255 5305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:57.472852 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 5225
I20260812 00:13:57.472990 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:13:57.473196 16783 external_mini_cluster.cc:1428] Setting key 3af409770c814948f209ac2ca61f215e
I20260812 00:13:57.473629 5305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:57.475433 5305 catalog_manager.cc:1347] Generated new cluster ID: 35eae5bc33114c6e8a8eb84e5e9098fa
I20260812 00:13:57.475493 5305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:13:57.480261 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.473860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.16.99.254:37367)
0812 00:13:57.474497 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.474505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.474530 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:13:57.474624 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.474630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.474692 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.474896 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:57.475158 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.476577 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.477760 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.478439 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.478909 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.479383 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:57.479386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:57.479387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:57.479597 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:57.479634 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:57.479649 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:57.479926 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.479956 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.480033 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":80,"threads_started":1}
I20260812 00:13:57.487984 5305 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:57.488796 5305 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:57.493523 5305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d: Generated new TSK 0
I20260812 00:13:57.493732 5305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:13:57.496599 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.193:0
--local_ip_for_outbound_sockets=127.16.99.193
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37367
--builtin_ntp_servers=127.16.99.212:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:57.603475 5317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:57.603709 5317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:57.603734 5317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:57.604943 5317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:57.604987 5317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:57.605014 5317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.193
I20260812 00:13:57.606544 5317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.99.193
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.193
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:57.606820 5317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:57.607131 5317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:57.613221 5322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.613226 5325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.613459 5323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:57.613706 5317 server_base.cc:1034] running on GCE node
I20260812 00:13:57.614013 5317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:57.614290 5317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:57.615497 5317 hybrid_clock.cc:648] HybridClock initialized: now 1786493637615481 us; error 31 us; skew 500 ppm
I20260812 00:13:57.617014 5317 webserver.cc:460] Webserver started at http://127.16.99.193:38487/ using document root <none> and password file <none>
I20260812 00:13:57.617309 5317 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:57.617384 5317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:57.617538 5317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:57.618764 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data/instance:
uuid: "2952cb4d7f714a2e8f23075ee1d1afb2"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "3205bd1975dd7aa94181acdcf7864def"
server_key_iv: "105ae8dbe1c596ba434f55aa66da69f7"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.619132 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance:
uuid: "2952cb4d7f714a2e8f23075ee1d1afb2"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "3205bd1975dd7aa94181acdcf7864def"
server_key_iv: "105ae8dbe1c596ba434f55aa66da69f7"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.620658 5317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:13:57.621621 5331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.621804 5317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:57.621912 5317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/wal
uuid: "2952cb4d7f714a2e8f23075ee1d1afb2"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "3205bd1975dd7aa94181acdcf7864def"
server_key_iv: "105ae8dbe1c596ba434f55aa66da69f7"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.622000 5317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:57.632354 5317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:57.632722 5317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:57.632938 5317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:57.633221 5317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:57.633625 5317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:57.633683 5317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.633728 5317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:57.633757 5317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.640319 5317 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.193:46835
I20260812 00:13:57.640415 5449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.193:46835 every 8 connection(s)
I20260812 00:13:57.640832 5317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/data/info.pb
I20260812 00:13:57.646468 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 5317
I20260812 00:13:57.646610 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-0/wal/instance
I20260812 00:13:57.646808 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.641689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.193:32863 (local address 127.16.99.254:37367)
0812 00:13:57.641768 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.641771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.641980 (+ 209us) server_negotiation.cc:407] Connection header received
0812 00:13:57.642701 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.642703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.642717 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.642776 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:57.643493 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.644107 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.645187 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.645396 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.645478 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.646481 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:57.646483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:57.646483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:57.646489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:57.646506 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:57.646517 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:57.646580 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.646696 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.646728 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 00:13:57.646890 16783 external_mini_cluster.cc:1428] Setting key 182f97335ff750836bab86f6ddac67c5
I20260812 00:13:57.647214 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.641646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37367 (local address 127.16.99.193:32863)
0812 00:13:57.641864 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.641902 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.642589 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.642835 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.642839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.642847 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:57.643386 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.643391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.644213 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.644215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.645066 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.645069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.645489 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.645490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.645639 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.646282 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:57.646283 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:57.646315 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:57.646385 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:57.646581 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:57.646588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:57.646591 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.646668 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.646784 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260812 00:13:57.647742 5450 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37367
I20260812 00:13:57.647837 5450 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:57.647993 5450 heartbeater.cc:511] Master 127.16.99.254:37367 requested a full tablet report, sending...
I20260812 00:13:57.648072 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.194:0
--local_ip_for_outbound_sockets=127.16.99.194
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37367
--builtin_ntp_servers=127.16.99.212:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 00:13:57.648514 5256 ts_manager.cc:194] Registered new tserver with Master: 2952cb4d7f714a2e8f23075ee1d1afb2 (127.16.99.193:46835)
I20260812 00:13:57.649164 5256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.193:32863
W20260812 00:13:57.734479 5453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:57.734706 5453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:57.734730 5453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:57.736117 5453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:57.736183 5453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:57.736218 5453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.194
I20260812 00:13:57.737788 5453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.99.194
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.194
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:57.738061 5453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:57.738418 5453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:57.743884 5461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.743870 5458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:57.743919 5453 server_base.cc:1034] running on GCE node
W20260812 00:13:57.743883 5459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:57.744534 5453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:57.744868 5453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:57.746039 5453 hybrid_clock.cc:648] HybridClock initialized: now 1786493637746021 us; error 34 us; skew 500 ppm
I20260812 00:13:57.747591 5453 webserver.cc:460] Webserver started at http://127.16.99.194:38011/ using document root <none> and password file <none>
I20260812 00:13:57.747927 5453 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:57.747979 5453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:57.748121 5453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:57.749321 5453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data/instance:
uuid: "8254cf2062f64825bcbb06eddce05b46"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "9e2f594899307ec43099bf83d11b68ba"
server_key_iv: "58ca64f78d02dd776e458a0b7be7b3a7"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.749701 5453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance:
uuid: "8254cf2062f64825bcbb06eddce05b46"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "9e2f594899307ec43099bf83d11b68ba"
server_key_iv: "58ca64f78d02dd776e458a0b7be7b3a7"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.751430 5453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:13:57.752367 5468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.752552 5453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:57.752682 5453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal
uuid: "8254cf2062f64825bcbb06eddce05b46"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "9e2f594899307ec43099bf83d11b68ba"
server_key_iv: "58ca64f78d02dd776e458a0b7be7b3a7"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.752746 5453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:57.768363 5453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:57.768687 5453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:57.768797 5453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:57.769001 5453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:57.769387 5453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:57.769426 5453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.769449 5453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:57.769518 5453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.776167 5453 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.194:45809
I20260812 00:13:57.776269 5586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.194:45809 every 8 connection(s)
I20260812 00:13:57.776638 5453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/data/info.pb
I20260812 00:13:57.782593 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.777465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.194:58307 (local address 127.16.99.254:37367)
0812 00:13:57.777514 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.777516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.777806 (+ 290us) server_negotiation.cc:407] Connection header received
0812 00:13:57.778443 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.778444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.778457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.778514 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:57.779263 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.779777 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.780994 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.781218 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.781300 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.782232 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:57.782234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:57.782234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:57.782240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:57.782256 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:57.782261 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:57.782333 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.782471 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.782504 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:13:57.782835 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.777440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37367 (local address 127.16.99.194:58307)
0812 00:13:57.777670 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.777709 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.778261 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.778597 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.778602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.778612 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:57.779146 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.779150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.779883 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.779884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.780891 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.780894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.781310 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.781311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.781452 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.782056 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:57.782057 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:57.782087 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:57.782118 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:57.782367 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:57.782380 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:57.782389 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.782474 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.782585 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260812 00:13:57.783367 5587 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37367
I20260812 00:13:57.783491 5587 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:57.783706 5587 heartbeater.cc:511] Master 127.16.99.254:37367 requested a full tablet report, sending...
I20260812 00:13:57.784140 5256 ts_manager.cc:194] Registered new tserver with Master: 8254cf2062f64825bcbb06eddce05b46 (127.16.99.194:45809)
I20260812 00:13:57.784509 5256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.194:58307
I20260812 00:13:57.784770 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 5453
I20260812 00:13:57.784897 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal/instance
I20260812 00:13:57.785178 16783 external_mini_cluster.cc:1428] Setting key b4057362b31a54ee1ab395a9fb314290
I20260812 00:13:57.786132 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.99.195:0
--local_ip_for_outbound_sockets=127.16.99.195
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37367
--builtin_ntp_servers=127.16.99.212:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 00:13:57.879295 5590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:57.879519 5590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:57.879544 5590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:57.880803 5590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:57.880851 5590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:57.880878 5590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.99.195
I20260812 00:13:57.882508 5590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.99.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.99.195
--webserver_port=0
--tserver_master_addrs=127.16.99.254:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.99.195
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:57.882794 5590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:57.883152 5590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:57.888784 5599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.888774 5596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:57.888779 5597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:57.889345 5590 server_base.cc:1034] running on GCE node
I20260812 00:13:57.889542 5590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:57.889788 5590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:57.890980 5590 hybrid_clock.cc:648] HybridClock initialized: now 1786493637890953 us; error 40 us; skew 500 ppm
I20260812 00:13:57.892736 5590 webserver.cc:460] Webserver started at http://127.16.99.195:41493/ using document root <none> and password file <none>
I20260812 00:13:57.893013 5590 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:57.893060 5590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:57.893174 5590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:57.894268 5590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data/instance:
uuid: "a10108fb27584af48d0559276d4617f5"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "4ded3e309dbeb0c1d9ae9563bc7b51d5"
server_key_iv: "95b4a7ae1792b33be1a69b2404ee8c3c"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.894693 5590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance:
uuid: "a10108fb27584af48d0559276d4617f5"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "4ded3e309dbeb0c1d9ae9563bc7b51d5"
server_key_iv: "95b4a7ae1792b33be1a69b2404ee8c3c"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.896163 5590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:57.897401 5605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.897809 5590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:57.897969 5590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal
uuid: "a10108fb27584af48d0559276d4617f5"
format_stamp: "Formatted at 2026-08-12 00:13:57 on dist-test-slave-zpfg"
server_key: "4ded3e309dbeb0c1d9ae9563bc7b51d5"
server_key_iv: "95b4a7ae1792b33be1a69b2404ee8c3c"
server_key_version: "encryptionkey@0"
I20260812 00:13:57.898077 5590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:57.907483 5590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:57.907917 5590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:57.908106 5590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:57.908406 5590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 00:13:57.908864 5590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 00:13:57.908926 5590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.908964 5590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 00:13:57.909015 5590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:57.915961 5590 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.195:37309
I20260812 00:13:57.916088 5721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.195:37309 every 8 connection(s)
I20260812 00:13:57.916445 5590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/data/info.pb
I20260812 00:13:57.922770 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.917358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:38569 (local address 127.16.99.254:37367)
0812 00:13:57.917462 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.917464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.917750 (+ 286us) server_negotiation.cc:407] Connection header received
0812 00:13:57.918414 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.918415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.918428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.918489 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:57.919243 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.919828 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.921109 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.921319 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.921383 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.922442 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:57.922443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:57.922443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:57.922449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:57.922464 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:57.922468 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:57.922517 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.922654 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.922689 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 00:13:57.922777 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 5590
I20260812 00:13:57.922928 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal/instance
I20260812 00:13:57.923022 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.917387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:37367 (local address 127.16.99.195:38569)
0812 00:13:57.917596 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.917637 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.918278 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.918562 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.918566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.918575 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:13:57.919116 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.919121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.919959 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.919960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.921002 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.921005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.921426 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.921427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.921568 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.922268 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 00:13:57.922269 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:13:57.922300 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 00:13:57.922327 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:13:57.922539 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:13:57.922542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:13:57.922546 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.922623 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.922753 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260812 00:13:57.923153 16783 external_mini_cluster.cc:1428] Setting key 67c7141ab7949aebf384bf4996517bff
I20260812 00:13:57.923841 5722 heartbeater.cc:347] Connected to a master server at 127.16.99.254:37367
I20260812 00:13:57.923952 5722 heartbeater.cc:464] Registering TS with master...
I20260812 00:13:57.924208 5722 heartbeater.cc:511] Master 127.16.99.254:37367 requested a full tablet report, sending...
I20260812 00:13:57.924690 5256 ts_manager.cc:194] Registered new tserver with Master: a10108fb27584af48d0559276d4617f5 (127.16.99.195:37309)
I20260812 00:13:57.925130 5256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.99.195:38569
I20260812 00:13:57.935537 16783 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 00:13:57.940631 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.937248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.16.99.254:37367)
0812 00:13:57.937315 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.937317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.937417 (+ 100us) server_negotiation.cc:407] Connection header received
0812 00:13:57.937470 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.937471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.937485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.937528 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:57.937747 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.938302 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.939171 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.939375 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.939441 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.940334 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:57.940335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:57.940335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:13:57.940343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:57.940362 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:13:57.940366 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:57.940427 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.940492 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.940522 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:13:57.942672 5256 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 00:13:57.944582 5256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 00:13:57.952443 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.947617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.16.99.194:45809)
0812 00:13:57.948011 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.948017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.948031 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:13:57.948175 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.948179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.948195 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.948279 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.948479 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.949643 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.951216 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.952036 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.952119 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.952220 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.952338 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.952372 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":97,"threads_started":1}
I20260812 00:13:57.952873 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.947479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.195:37309 (local address 127.0.0.1:44816)
0812 00:13:57.947906 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.947952 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.948473 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.948667 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.948668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.948670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.948717 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.948719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.949651 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.949652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.951116 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.951119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.952637 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.952638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.952697 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.952701 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.952758 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.952804 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":59,"thread_start_us":85,"threads_started":1}
I20260812 00:13:57.953078 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.947851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.16.99.193:46835)
0812 00:13:57.948047 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.948052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.948264 (+ 212us) server_negotiation.cc:407] Connection header received
0812 00:13:57.948312 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.948315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.948331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.948405 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.948671 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.949836 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.951988 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.952767 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.952824 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.952911 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.952946 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.952995 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260812 00:13:57.953174 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.947747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.16.99.195:37309)
0812 00:13:57.948454 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.948458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.948470 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:13:57.948553 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.948555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.948567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.948653 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.948861 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.949562 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.951274 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.952036 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.952119 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.952220 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.953031 (+ 811us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.953078 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":92,"threads_started":1}
I20260812 00:13:57.953068 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.947740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46835 (local address 127.0.0.1:55450)
0812 00:13:57.948100 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.948110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.948174 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.948556 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.948557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.948559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.948598 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.948600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.951229 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.951229 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.951888 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.951890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.952840 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.952840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.952900 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.952903 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.952935 (+ 32us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.952996 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":98,"threads_started":1}
I20260812 00:13:57.952463 5735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.947479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:45809 (local address 127.0.0.1:43128)
0812 00:13:57.947906 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.947954 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.948036 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.948302 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.948306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.948308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.948391 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.948394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.949714 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.949715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.951116 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.951119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.952126 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.952127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.952209 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.952234 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.952313 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.952365 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":50,"thread_start_us":89,"threads_started":1}
I20260812 00:13:57.954021 5379 tablet_service.cc:1467] Processing CreateTablet for tablet d4a6d6c6648444f9ada5aad178a20bdf (DEFAULT_TABLE table=test-table [id=d091267841cd495ca0b28eeea87a580c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:57.954097 5518 tablet_service.cc:1467] Processing CreateTablet for tablet d4a6d6c6648444f9ada5aad178a20bdf (DEFAULT_TABLE table=test-table [id=d091267841cd495ca0b28eeea87a580c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:57.954387 5379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a6d6c6648444f9ada5aad178a20bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:57.954483 5518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a6d6c6648444f9ada5aad178a20bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:57.957337 5738 tablet_bootstrap.cc:492] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2: Bootstrap starting.
I20260812 00:13:57.957366 5739 tablet_bootstrap.cc:492] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Bootstrap starting.
I20260812 00:13:57.957988 5739 tablet_bootstrap.cc:654] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:57.957875 5653 tablet_service.cc:1467] Processing CreateTablet for tablet d4a6d6c6648444f9ada5aad178a20bdf (DEFAULT_TABLE table=test-table [id=d091267841cd495ca0b28eeea87a580c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 00:13:57.958158 5738 tablet_bootstrap.cc:654] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:57.958259 5653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a6d6c6648444f9ada5aad178a20bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:57.958334 5739 log.cc:824] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:57.958561 5738 log.cc:824] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:57.959481 5738 tablet_bootstrap.cc:492] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2: No bootstrap required, opened a new log
I20260812 00:13:57.959570 5738 ts_tablet_manager.cc:1397] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 00:13:57.961372 5742 tablet_bootstrap.cc:492] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Bootstrap starting.
I20260812 00:13:57.961629 5738 raft_consensus.cc:348] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.961807 5738 raft_consensus.cc:374] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:57.961848 5738 raft_consensus.cc:729] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2952cb4d7f714a2e8f23075ee1d1afb2, State: Initialized, Role: FOLLOWER
I20260812 00:13:57.961951 5739 tablet_bootstrap.cc:492] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: No bootstrap required, opened a new log
I20260812 00:13:57.962005 5742 tablet_bootstrap.cc:654] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:57.961972 5738 consensus_queue.cc:260] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.962018 5739 ts_tablet_manager.cc:1397] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 00:13:57.962261 5738 ts_tablet_manager.cc:1428] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 00:13:57.962313 5450 heartbeater.cc:503] Master 127.16.99.254:37367 was elected leader, sending a full tablet report...
I20260812 00:13:57.962345 5742 log.cc:824] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:57.963145 5742 tablet_bootstrap.cc:492] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: No bootstrap required, opened a new log
I20260812 00:13:57.963215 5742 ts_tablet_manager.cc:1397] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 00:13:57.964828 5739 raft_consensus.cc:348] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.965001 5739 raft_consensus.cc:374] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:57.965044 5739 raft_consensus.cc:729] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8254cf2062f64825bcbb06eddce05b46, State: Initialized, Role: FOLLOWER
I20260812 00:13:57.965178 5739 consensus_queue.cc:260] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.965454 5739 ts_tablet_manager.cc:1428] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260812 00:13:57.965514 5587 heartbeater.cc:503] Master 127.16.99.254:37367 was elected leader, sending a full tablet report...
I20260812 00:13:57.965863 5742 raft_consensus.cc:348] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.966019 5742 raft_consensus.cc:374] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:57.966076 5742 raft_consensus.cc:729] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10108fb27584af48d0559276d4617f5, State: Initialized, Role: FOLLOWER
I20260812 00:13:57.966190 5742 consensus_queue.cc:260] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.966511 5742 ts_tablet_manager.cc:1428] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260812 00:13:57.966527 5722 heartbeater.cc:503] Master 127.16.99.254:37367 was elected leader, sending a full tablet report...
I20260812 00:13:57.985881 5747 raft_consensus.cc:482] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 00:13:57.986014 5747 raft_consensus.cc:504] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.986557 5747 leader_election.cc:288] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8254cf2062f64825bcbb06eddce05b46 (127.16.99.194:45809), 2952cb4d7f714a2e8f23075ee1d1afb2 (127.16.99.193:46835)
I20260812 00:13:57.989827 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.986749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46835 (local address 127.16.99.195:40479)
0812 00:13:57.986822 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.986838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.986902 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.987060 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.987061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.987063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.987256 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.987260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.987994 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.987995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.988942 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.988945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.989569 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.989570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.989658 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.989663 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.989725 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.989765 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260812 00:13:57.989918 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.986845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:40479 (local address 127.16.99.193:46835)
0812 00:13:57.986905 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.986907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.986919 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:13:57.986961 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.986962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.986973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.987039 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.987356 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.987906 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.989042 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.989476 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.989542 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.989550 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.989786 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.989841 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:13:57.990005 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.986778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.194:45809 (local address 127.16.99.195:52391)
0812 00:13:57.986958 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:57.986967 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:13:57.987031 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:13:57.987213 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:13:57.987214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:13:57.987216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 00:13:57.987287 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.987321 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.988031 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.988036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.988952 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:13:57.988955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.989661 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.989661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:13:57.989812 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.989816 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 00:13:57.989878 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 00:13:57.989913 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":84,"thread_start_us":117,"threads_started":1}
I20260812 00:13:57.990057 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:57.986850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.99.195:52391 (local address 127.16.99.194:45809)
0812 00:13:57.986898 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:57.986901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:57.987044 (+ 143us) server_negotiation.cc:407] Connection header received
0812 00:13:57.987093 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:57.987094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:57.987108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:57.987166 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 00:13:57.987415 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.987916 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.989090 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:57.989557 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:57.989648 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:57.989656 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:57.989905 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 00:13:57.989957 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:13:57.990216 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a6d6c6648444f9ada5aad178a20bdf" candidate_uuid: "a10108fb27584af48d0559276d4617f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" is_pre_election: true
I20260812 00:13:57.990404 5400 raft_consensus.cc:2454] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a10108fb27584af48d0559276d4617f5 in term 0.
I20260812 00:13:57.990406 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a6d6c6648444f9ada5aad178a20bdf" candidate_uuid: "a10108fb27584af48d0559276d4617f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8254cf2062f64825bcbb06eddce05b46" is_pre_election: true
I20260812 00:13:57.990567 5538 raft_consensus.cc:2454] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a10108fb27584af48d0559276d4617f5 in term 0.
I20260812 00:13:57.990660 5607 leader_election.cc:302] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2952cb4d7f714a2e8f23075ee1d1afb2, a10108fb27584af48d0559276d4617f5; no voters:
I20260812 00:13:57.990806 5747 raft_consensus.cc:2781] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 00:13:57.990868 5747 raft_consensus.cc:482] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 00:13:57.990899 5747 raft_consensus.cc:3037] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:57.991695 5747 raft_consensus.cc:504] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.991855 5747 leader_election.cc:288] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [CANDIDATE]: Term 1 election: Requested vote from peers 8254cf2062f64825bcbb06eddce05b46 (127.16.99.194:45809), 2952cb4d7f714a2e8f23075ee1d1afb2 (127.16.99.193:46835)
I20260812 00:13:57.992049 5538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a6d6c6648444f9ada5aad178a20bdf" candidate_uuid: "a10108fb27584af48d0559276d4617f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8254cf2062f64825bcbb06eddce05b46"
I20260812 00:13:57.992163 5538 raft_consensus.cc:3037] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:57.992215 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a6d6c6648444f9ada5aad178a20bdf" candidate_uuid: "a10108fb27584af48d0559276d4617f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2"
I20260812 00:13:57.992347 5400 raft_consensus.cc:3037] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:57.993227 5538 raft_consensus.cc:2454] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10108fb27584af48d0559276d4617f5 in term 1.
I20260812 00:13:57.993360 5400 raft_consensus.cc:2454] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10108fb27584af48d0559276d4617f5 in term 1.
I20260812 00:13:57.993432 5609 leader_election.cc:302] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8254cf2062f64825bcbb06eddce05b46, a10108fb27584af48d0559276d4617f5; no voters:
I20260812 00:13:57.993584 5747 raft_consensus.cc:2781] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:57.993757 5747 raft_consensus.cc:686] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 1 LEADER]: Becoming Leader. State: Replica: a10108fb27584af48d0559276d4617f5, State: Running, Role: LEADER
I20260812 00:13:57.993897 5747 consensus_queue.cc:237] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } }
I20260812 00:13:57.994894 5256 catalog_manager.cc:5495] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a10108fb27584af48d0559276d4617f5 (127.16.99.195). New cstate: current_term: 1 leader_uuid: "a10108fb27584af48d0559276d4617f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10108fb27584af48d0559276d4617f5" member_type: VOTER last_known_addr { host: "127.16.99.195" port: 37309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 } health_report { overall_health: UNKNOWN } } }
I20260812 00:13:58.011152 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:58.008384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.16.99.195:37309)
0812 00:13:58.008451 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:58.008453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:58.008462 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:13:58.008504 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:58.008505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:58.008524 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:58.008596 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:13:58.008837 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:58.009419 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:58.010314 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:58.010578 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:58.010641 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:58.010828 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:13:58.010938 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:13:58.011004 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:58.011039 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 00:13:58.011056 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 00:13:58.013201 5538 raft_consensus.cc:1261] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 1 FOLLOWER]: Refusing update from remote peer a10108fb27584af48d0559276d4617f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:58.013226 5400 raft_consensus.cc:1261] T d4a6d6c6648444f9ada5aad178a20bdf P 2952cb4d7f714a2e8f23075ee1d1afb2 [term 1 FOLLOWER]: Refusing update from remote peer a10108fb27584af48d0559276d4617f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:13:58.013693 5750 consensus_queue.cc:1035] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2952cb4d7f714a2e8f23075ee1d1afb2" member_type: VOTER last_known_addr { host: "127.16.99.193" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:58.013862 5747 consensus_queue.cc:1035] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8254cf2062f64825bcbb06eddce05b46" member_type: VOTER last_known_addr { host: "127.16.99.194" port: 45809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:13:58.017117 5753 mvcc.cc:205] Tried to move back new op lower bound from 7317477941299109888 to 7317477941223931904. Current Snapshot: MvccSnapshot[applied={T|T < 7317477941299109888}]
I20260812 00:13:58.017621 5754 mvcc.cc:205] Tried to move back new op lower bound from 7317477941299109888 to 7317477941223931904. Current Snapshot: MvccSnapshot[applied={T|T < 7317477941299109888}]
I20260812 00:13:58.017756 5755 mvcc.cc:205] Tried to move back new op lower bound from 7317477941299109888 to 7317477941223931904. Current Snapshot: MvccSnapshot[applied={T|T < 7317477941299109888}]
I20260812 00:13:58.020489 5255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:58.020594 5255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 00:13:58.021548 5255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc1304a851a0412d9bbb6c91d1c7552d: Sending DeleteTablet for 3 replicas of tablet d4a6d6c6648444f9ada5aad178a20bdf
I20260812 00:13:58.021951 5379 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a6d6c6648444f9ada5aad178a20bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:58 UTC) from {username='slave'} at 127.0.0.1:55450
I20260812 00:13:58.021979 5518 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a6d6c6648444f9ada5aad178a20bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:58 UTC) from {username='slave'} at 127.0.0.1:43128
I20260812 00:13:58.022370 5653 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a6d6c6648444f9ada5aad178a20bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 00:13:58 UTC) from {username='slave'} at 127.0.0.1:44816
I20260812 00:13:58.022451 5763 tablet_replica.cc:330] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: stopping tablet replica
I20260812 00:13:58.022516 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 5317
I20260812 00:13:58.022580 5763 raft_consensus.cc:2229] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 00:13:58.022707 5763 raft_consensus.cc:2258] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:58.025969 5763 ts_tablet_manager.cc:1905] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:58.028406 5763 ts_tablet_manager.cc:1918] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:58.028518 5763 log.cc:1196] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-1/wal/wals/d4a6d6c6648444f9ada5aad178a20bdf
I20260812 00:13:58.028817 5763 ts_tablet_manager.cc:1939] T d4a6d6c6648444f9ada5aad178a20bdf P 8254cf2062f64825bcbb06eddce05b46: Deleting consensus metadata
I20260812 00:13:58.029368 5243 catalog_manager.cc:4841] TS 8254cf2062f64825bcbb06eddce05b46 (127.16.99.194:45809): tablet d4a6d6c6648444f9ada5aad178a20bdf (table test-table [id=d091267841cd495ca0b28eeea87a580c]) successfully deleted
W20260812 00:13:58.031131 5241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.99.193:46835 (error 108)
W20260812 00:13:58.031831 5735 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:13:58.031627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.193:46835 (local address 127.0.0.1:55456)
0812 00:13:58.031718 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 00:13:58.031774 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46835: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 00:13:58.032037 5241 catalog_manager.cc:4584] TS 2952cb4d7f714a2e8f23075ee1d1afb2 (127.16.99.193:46835): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4a6d6c6648444f9ada5aad178a20bdf: Network error: Client connection negotiation failed: client connection to 127.16.99.193:46835: connect: Connection reset by peer (error 104)
I20260812 00:13:58.032153 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 5453
I20260812 00:13:58.038511 5765 tablet_replica.cc:330] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: stopping tablet replica
I20260812 00:13:58.038687 5765 raft_consensus.cc:2229] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 00:13:58.038841 5765 raft_consensus.cc:2258] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 00:13:58.039461 5765 ts_tablet_manager.cc:1905] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 00:13:58.041401 5765 ts_tablet_manager.cc:1918] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 00:13:58.041479 5765 log.cc:1196] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Deleting WAL directory at /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786493381608766-16783-0/minicluster-data/ts-2/wal/wals/d4a6d6c6648444f9ada5aad178a20bdf
I20260812 00:13:58.041747 5765 ts_tablet_manager.cc:1939] T d4a6d6c6648444f9ada5aad178a20bdf P a10108fb27584af48d0559276d4617f5: Deleting consensus metadata
I20260812 00:13:58.042630 5241 catalog_manager.cc:4841] TS a10108fb27584af48d0559276d4617f5 (127.16.99.195:37309): tablet d4a6d6c6648444f9ada5aad178a20bdf (table test-table [id=d091267841cd495ca0b28eeea87a580c]) successfully deleted
I20260812 00:13:58.047008 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 5590
I20260812 00:13:58.054179 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 5225
2026-08-12T00:13:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (746 ms)
[----------] 6 tests from EncryptionPolicyTest (9422 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:13:58 dist-test-slave-zpfg krb5kdc[5769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:13:58 dist-test-slave-zpfg krb5kdc[5769](info): set up 2 sockets
Aug 12 00:13:58 dist-test-slave-zpfg krb5kdc[5769](info): commencing operation
krb5kdc: starting...
W20260812 00:13:59.219426 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:13:59 dist-test-slave-zpfg krb5kdc[5769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:13:59.353765 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:13:59.439970 5782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:13:59.440205 5782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:13:59.440232 5782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:13:59.441501 5782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:13:59.441552 5782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:13:59.441567 5782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:13:59.441581 5782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:13:59.441592 5782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:13:59.443205 5782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44745
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:13:59.443492 5782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:13:59.443754 5782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:13:59.449088 5787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:59.449150 5782 server_base.cc:1034] running on GCE node
W20260812 00:13:59.449076 5788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:13:59.449076 5790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:13:59.449836 5782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:13:59.450125 5782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:13:59.451293 5782 hybrid_clock.cc:648] HybridClock initialized: now 1786493639451271 us; error 47 us; skew 500 ppm
Aug 12 00:13:59 dist-test-slave-zpfg krb5kdc[5769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:13:59.459863 5782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:13:59.460361 5782 webserver.cc:460] Webserver started at http://127.0.0.1:34881/ using document root <none> and password file <none>
I20260812 00:13:59.460580 5782 fs_manager.cc:359] Metadata directory not provided
I20260812 00:13:59.460624 5782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:13:59.460733 5782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:13:59.461841 5782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "d73105f030314c1b9d2d288b0682091d"
format_stamp: "Formatted at 2026-08-12 00:13:59 on dist-test-slave-zpfg"
server_key: "ee69ce554ee550b78cff292a05fbb78f"
server_key_iv: "faa03d24c6900eb9121869ab03af416f"
server_key_version: "encryptionkey@0"
I20260812 00:13:59.462165 5782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "d73105f030314c1b9d2d288b0682091d"
format_stamp: "Formatted at 2026-08-12 00:13:59 on dist-test-slave-zpfg"
server_key: "ee69ce554ee550b78cff292a05fbb78f"
server_key_iv: "faa03d24c6900eb9121869ab03af416f"
server_key_version: "encryptionkey@0"
I20260812 00:13:59.463624 5782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 00:13:59.464443 5797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:13:59.464681 5782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:13:59.464797 5782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "d73105f030314c1b9d2d288b0682091d"
format_stamp: "Formatted at 2026-08-12 00:13:59 on dist-test-slave-zpfg"
server_key: "ee69ce554ee550b78cff292a05fbb78f"
server_key_iv: "faa03d24c6900eb9121869ab03af416f"
server_key_version: "encryptionkey@0"
I20260812 00:13:59.464881 5782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:13:59.476326 5782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:13:59.479506 5782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:13:59.479696 5782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:13:59.484609 5782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44745
I20260812 00:13:59.484679 5854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44745 every 8 connection(s)
I20260812 00:13:59.485114 5782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:13:59.486083 5855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:13:59.488783 5855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d: Bootstrap starting.
I20260812 00:13:59.489476 5855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d: Neither blocks nor log segments found. Creating new log.
I20260812 00:13:59.489799 5855 log.cc:824] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:13:59.490199 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 5782
I20260812 00:13:59.490291 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:13:59.490512 16783 external_mini_cluster.cc:1428] Setting key c443e47f64cf7a9da6d503002fd19da5
I20260812 00:13:59.490924 5855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d: No bootstrap required, opened a new log
I20260812 00:13:59.493237 5855 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73105f030314c1b9d2d288b0682091d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260812 00:13:59.493464 5855 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:13:59.493513 5855 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73105f030314c1b9d2d288b0682091d, State: Initialized, Role: FOLLOWER
I20260812 00:13:59.493654 5855 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73105f030314c1b9d2d288b0682091d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260812 00:13:59.493777 5855 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:13:59.493815 5855 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:13:59.493858 5855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:13:59.494786 5855 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73105f030314c1b9d2d288b0682091d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260812 00:13:59.494910 5855 leader_election.cc:302] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73105f030314c1b9d2d288b0682091d; no voters:
I20260812 00:13:59.495090 5855 leader_election.cc:288] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:13:59.495312 5861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:13:59.495383 5855 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:13:59.495605 5861 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [term 1 LEADER]: Becoming Leader. State: Replica: d73105f030314c1b9d2d288b0682091d, State: Running, Role: LEADER
I20260812 00:13:59.495767 5861 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73105f030314c1b9d2d288b0682091d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } }
I20260812 00:13:59.496482 5863 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73105f030314c1b9d2d288b0682091d. Latest consensus state: current_term: 1 leader_uuid: "d73105f030314c1b9d2d288b0682091d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73105f030314c1b9d2d288b0682091d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } } }
I20260812 00:13:59.496595 5863 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:59.496649 5862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73105f030314c1b9d2d288b0682091d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73105f030314c1b9d2d288b0682091d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44745 } } }
I20260812 00:13:59.496711 5862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d [sys.catalog]: This master's current role is: LEADER
I20260812 00:13:59.496953 5874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:13:59.497640 5874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:13:59.499682 5874 catalog_manager.cc:1347] Generated new cluster ID: 22eec184a5594fbda0000f4b74f947a2
I20260812 00:13:59.499898 5874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:13:59 dist-test-slave-zpfg krb5kdc[5769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:13:59.505282 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:59.491179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40649 (local address 127.0.0.1:44745)
0812 00:13:59.491428 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:59.491439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:59.491478 (+ 39us) server_negotiation.cc:407] Connection header received
0812 00:13:59.492390 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:59.492400 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:59.492477 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:59.492645 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:59.493346 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:59.495175 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:59.496426 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:59.497218 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:59.497353 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:59.500663 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:59.500669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:59.500670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:59.500685 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:59.503066 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:59.503656 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:59.503660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:59.503661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:59.503809 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:59.504112 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:59.504113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:59.504113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:59.504473 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:59.504657 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:59.504892 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 00:13:59.505012 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260812 00:13:59.506487 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:13:59.511804 5874 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:13:59.512583 5874 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:13:59.519979 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:13:59.510735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52007 (local address 127.0.0.1:44745)
0812 00:13:59.510777 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:13:59.510779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:13:59.511365 (+ 586us) server_negotiation.cc:407] Connection header received
0812 00:13:59.511453 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:13:59.511455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:13:59.511466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:13:59.511532 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:13:59.512347 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:59.513178 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:59.514334 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:13:59.514738 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:13:59.514817 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:13:59.516489 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:13:59.516491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:13:59.516491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:13:59.516499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:13:59.518414 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:59.518962 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:59.518963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:59.518964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:59.519037 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:13:59.519296 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:13:59.519297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:13:59.519297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:13:59.519465 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:13:59.519608 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 00:13:59.519775 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 00:13:59.519867 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":18}
I20260812 00:13:59.520689 5874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73105f030314c1b9d2d288b0682091d: Generated new TSK 0
I20260812 00:13:59.520890 5874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:13:59.528657 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 5782
W20260812 00:13:59.536753 5767 connection.cc:535] client connection to 127.0.0.1:44745 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44745): Connection reset by peer (error 104)
2026-08-12T00:13:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:13:59 dist-test-slave-zpfg krb5kdc[5888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:13:59 dist-test-slave-zpfg krb5kdc[5888](info): set up 2 sockets
Aug 12 00:13:59 dist-test-slave-zpfg krb5kdc[5888](info): commencing operation
krb5kdc: starting...
W20260812 00:14:00.654747 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:00 dist-test-slave-zpfg krb5kdc[5888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:00.800587 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:00.887760 5903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:00.888003 5903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:00.888031 5903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:00.889284 5903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:00.889341 5903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:00.889356 5903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:00.889369 5903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:00.889381 5903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:00.891072 5903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:00.891371 5903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:00.891652 5903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:00.897507 5908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:00.897508 5911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:00.897552 5903 server_base.cc:1034] running on GCE node
W20260812 00:14:00.897507 5909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:00.898437 5903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:00.898866 5903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:00.900089 5903 hybrid_clock.cc:648] HybridClock initialized: now 1786493640900064 us; error 71 us; skew 500 ppm
Aug 12 00:14:00 dist-test-slave-zpfg krb5kdc[5888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:00.903548 5903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:00.904070 5903 webserver.cc:460] Webserver started at http://127.0.0.1:46461/ using document root <none> and password file <none>
I20260812 00:14:00.904340 5903 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:00.904388 5903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:00.904549 5903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:00.905764 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "0df4a27ead2e4bc08664cc0c6958a48f"
format_stamp: "Formatted at 2026-08-12 00:14:00 on dist-test-slave-zpfg"
server_key: "d0a942f6a38e5d480acb04de2336574b"
server_key_iv: "da58af21a3571afa42a1f0087851b0d3"
server_key_version: "encryptionkey@0"
I20260812 00:14:00.906185 5903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "0df4a27ead2e4bc08664cc0c6958a48f"
format_stamp: "Formatted at 2026-08-12 00:14:00 on dist-test-slave-zpfg"
server_key: "d0a942f6a38e5d480acb04de2336574b"
server_key_iv: "da58af21a3571afa42a1f0087851b0d3"
server_key_version: "encryptionkey@0"
I20260812 00:14:00.907760 5903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:00.908730 5918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:00.908962 5903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 00:14:00.909089 5903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "0df4a27ead2e4bc08664cc0c6958a48f"
format_stamp: "Formatted at 2026-08-12 00:14:00 on dist-test-slave-zpfg"
server_key: "d0a942f6a38e5d480acb04de2336574b"
server_key_iv: "da58af21a3571afa42a1f0087851b0d3"
server_key_version: "encryptionkey@0"
I20260812 00:14:00.909158 5903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:00.925208 5903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:00.928381 5903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:00.928619 5903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:00.933832 5903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35121
I20260812 00:14:00.933866 5974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35121 every 8 connection(s)
I20260812 00:14:00.934419 5903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:00.935482 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:00.937577 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 5903
I20260812 00:14:00.937731 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:00.937964 16783 external_mini_cluster.cc:1428] Setting key fa8368dc89a4776220e12ef4091c7d61
I20260812 00:14:00.938906 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f: Bootstrap starting.
I20260812 00:14:00.939633 5975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:00.940088 5975 log.cc:824] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:00.941038 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f: No bootstrap required, opened a new log
I20260812 00:14:00.943877 5975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35121 } }
I20260812 00:14:00.944119 5975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:00.944168 5975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df4a27ead2e4bc08664cc0c6958a48f, State: Initialized, Role: FOLLOWER
I20260812 00:14:00.944329 5975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35121 } }
I20260812 00:14:00.944410 5975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:00.944437 5975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:00.944509 5975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:00.945212 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:00.938764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46243 (local address 127.0.0.1:35121)
0812 00:14:00.938963 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:00.938972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:00.938999 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:14:00.939514 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:00.939522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:00.939585 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:00.939722 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:00.940108 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:00.941495 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:00.942845 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:00.943317 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:00.943408 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:00.944362 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:00.944365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:00.944366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:00.944638 (+ 272us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:00.944684 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:00.944697 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:00.944781 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:00.944954 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 00:14:00.944992 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260812 00:14:00.945426 5975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35121 } }
I20260812 00:14:00.945549 5975 leader_election.cc:302] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df4a27ead2e4bc08664cc0c6958a48f; no voters:
I20260812 00:14:00.945778 5975 leader_election.cc:288] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:00.945812 5980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:00.946064 5980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [term 1 LEADER]: Becoming Leader. State: Replica: 0df4a27ead2e4bc08664cc0c6958a48f, State: Running, Role: LEADER
I20260812 00:14:00.946133 5975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:00.946302 5980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35121 } }
I20260812 00:14:00.946447 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:00.946799 5981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35121 } } }
I20260812 00:14:00.946913 5981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:00.947136 5982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df4a27ead2e4bc08664cc0c6958a48f. Latest consensus state: current_term: 1 leader_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df4a27ead2e4bc08664cc0c6958a48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35121 } } }
I20260812 00:14:00.947211 5982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:00.947188 5989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:00.948031 5989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:00.949851 5989 catalog_manager.cc:1347] Generated new cluster ID: 9b79d01366a241a98181243371d430d3
I20260812 00:14:00.949919 5989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:00.954098 5989 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:00.954808 5989 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:00.958365 5978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:00.950816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44187 (local address 127.0.0.1:35121)
0812 00:14:00.950867 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:00.950871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:00.953622 (+ 2751us) server_negotiation.cc:407] Connection header received
0812 00:14:00.953717 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:00.953718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:00.953731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:00.953792 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:00.954096 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:00.955466 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:00.956706 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:00.957082 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:00.957171 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:00.958022 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:00.958024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:00.958024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:00.958031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:00.958050 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:00.958056 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:00.958124 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:00.958216 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 00:14:00.958239 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":24}
I20260812 00:14:00.961607 5989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df4a27ead2e4bc08664cc0c6958a48f: Generated new TSK 0
I20260812 00:14:00.961858 5989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:00.968645 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 5903
W20260812 00:14:00.975703 5886 connection.cc:535] client connection to 127.0.0.1:35121 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35121): Connection reset by peer (error 104)
2026-08-12T00:14:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:01 dist-test-slave-zpfg krb5kdc[6006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:01 dist-test-slave-zpfg krb5kdc[6006](info): set up 2 sockets
Aug 12 00:14:01 dist-test-slave-zpfg krb5kdc[6006](info): commencing operation
krb5kdc: starting...
W20260812 00:14:02.131276 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:02 dist-test-slave-zpfg krb5kdc[6006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:02.274075 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:02.377084 6020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:02.377355 6020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:02.377396 6020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:02.378752 6020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:02.378826 6020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:02.378850 6020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:02.378872 6020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:02.378902 6020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:02.380504 6020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:02.380825 6020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:02.381072 6020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:14:02.386569 6020 server_base.cc:1034] running on GCE node
W20260812 00:14:02.386512 6027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:02.386538 6025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:02.386586 6029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:02.387035 6020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:02.387387 6020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:02.388571 6020 hybrid_clock.cc:648] HybridClock initialized: now 1786493642388566 us; error 41 us; skew 500 ppm
Aug 12 00:14:02 dist-test-slave-zpfg krb5kdc[6006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:02.391482 6020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:02.391994 6020 webserver.cc:460] Webserver started at http://127.0.0.1:41323/ using document root <none> and password file <none>
I20260812 00:14:02.392266 6020 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:02.392315 6020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:02.392446 6020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:02.393563 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "cd006e18489a4c79a3ae8460f1d3737a"
format_stamp: "Formatted at 2026-08-12 00:14:02 on dist-test-slave-zpfg"
server_key: "54bfbd1bd89dc7a79ac6ca65ee8e0a19"
server_key_iv: "9dda82e242d89d9efc55ca540ecbbdaa"
server_key_version: "encryptionkey@0"
I20260812 00:14:02.393940 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "cd006e18489a4c79a3ae8460f1d3737a"
format_stamp: "Formatted at 2026-08-12 00:14:02 on dist-test-slave-zpfg"
server_key: "54bfbd1bd89dc7a79ac6ca65ee8e0a19"
server_key_iv: "9dda82e242d89d9efc55ca540ecbbdaa"
server_key_version: "encryptionkey@0"
I20260812 00:14:02.395494 6020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:14:02.396370 6036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:02.396683 6020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:02.396770 6020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "cd006e18489a4c79a3ae8460f1d3737a"
format_stamp: "Formatted at 2026-08-12 00:14:02 on dist-test-slave-zpfg"
server_key: "54bfbd1bd89dc7a79ac6ca65ee8e0a19"
server_key_iv: "9dda82e242d89d9efc55ca540ecbbdaa"
server_key_version: "encryptionkey@0"
I20260812 00:14:02.396869 6020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:02.400607 6020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:02.403136 6020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:02.403296 6020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:02.407555 6020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37479
I20260812 00:14:02.407590 6091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37479 every 8 connection(s)
I20260812 00:14:02.407961 6020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:02.408916 6092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:02.410557 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6020
I20260812 00:14:02.410691 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:02.410892 16783 external_mini_cluster.cc:1428] Setting key 7e959731f2b7ed8db0ece04fc4a42033
I20260812 00:14:02.411778 6092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a: Bootstrap starting.
I20260812 00:14:02.412421 6092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:02.412801 6092 log.cc:824] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:02.413638 6092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a: No bootstrap required, opened a new log
I20260812 00:14:02.413657 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:02.411542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51065 (local address 127.0.0.1:37479)
0812 00:14:02.411728 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:02.411739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:02.411774 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:14:02.412245 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:02.412251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:02.412279 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:02.412397 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:02.413209 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:02.413212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:02.413212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:02.413217 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:02.413255 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:02.413265 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:02.413364 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:02.413425 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 00:14:02.413467 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260812 00:14:02.414850 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:02.416538 6092 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006e18489a4c79a3ae8460f1d3737a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37479 } }
I20260812 00:14:02.416745 6092 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:02.416793 6092 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd006e18489a4c79a3ae8460f1d3737a, State: Initialized, Role: FOLLOWER
I20260812 00:14:02.417006 6092 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006e18489a4c79a3ae8460f1d3737a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37479 } }
I20260812 00:14:02.417186 6092 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:02.417256 6092 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:02.417316 6092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:02.417837 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:02.416628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59269 (local address 127.0.0.1:37479)
0812 00:14:02.416672 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:02.416674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:02.416773 (+ 99us) server_negotiation.cc:407] Connection header received
0812 00:14:02.416842 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:02.416843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:02.416848 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:02.416908 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:02.417581 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:02.417582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:02.417583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:02.417586 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:02.417599 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:02.417602 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:02.417647 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:02.417699 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 00:14:02.417711 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:14:02.418159 6092 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006e18489a4c79a3ae8460f1d3737a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37479 } }
I20260812 00:14:02.418270 6092 leader_election.cc:302] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd006e18489a4c79a3ae8460f1d3737a; no voters:
I20260812 00:14:02.418468 6092 leader_election.cc:288] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:02.418512 6102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:02.418664 6102 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [term 1 LEADER]: Becoming Leader. State: Replica: cd006e18489a4c79a3ae8460f1d3737a, State: Running, Role: LEADER
I20260812 00:14:02.418756 6102 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006e18489a4c79a3ae8460f1d3737a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37479 } }
I20260812 00:14:02.418761 6092 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:02.419128 6104 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd006e18489a4c79a3ae8460f1d3737a. Latest consensus state: current_term: 1 leader_uuid: "cd006e18489a4c79a3ae8460f1d3737a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006e18489a4c79a3ae8460f1d3737a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37479 } } }
I20260812 00:14:02.419138 6103 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd006e18489a4c79a3ae8460f1d3737a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd006e18489a4c79a3ae8460f1d3737a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37479 } } }
I20260812 00:14:02.419239 6104 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:02.419253 6103 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:02.419447 6109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:02.419968 6109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:02.421542 6109 catalog_manager.cc:1347] Generated new cluster ID: 90b543b9784d4f6886d3910d2a51ca0b
I20260812 00:14:02.421592 6109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:02.431607 6109 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:02.431840 6109 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:02.438630 6109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd006e18489a4c79a3ae8460f1d3737a: Generated new TSK 0
I20260812 00:14:02.438874 6109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:02.444235 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6020
2026-08-12T00:14:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:02 dist-test-slave-zpfg krb5kdc[6124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:02 dist-test-slave-zpfg krb5kdc[6124](info): set up 2 sockets
Aug 12 00:14:02 dist-test-slave-zpfg krb5kdc[6124](info): commencing operation
krb5kdc: starting...
W20260812 00:14:03.562842 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:03 dist-test-slave-zpfg krb5kdc[6124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:03.706964 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:03.801246 6138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:03.801553 6138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:03.801611 6138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:03.803129 6138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:03.803213 6138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:03.803241 6138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:03.803262 6138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:03.803280 6138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:03.805033 6138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:03.805367 6138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:03.805639 6138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:03.811295 6144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:03.811498 6146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:03.811563 6138 server_base.cc:1034] running on GCE node
W20260812 00:14:03.811493 6143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:03.811906 6138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:03.812242 6138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:03.813480 6138 hybrid_clock.cc:648] HybridClock initialized: now 1786493643813453 us; error 38 us; skew 500 ppm
Aug 12 00:14:03 dist-test-slave-zpfg krb5kdc[6124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:03.816551 6138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:03.817080 6138 webserver.cc:460] Webserver started at http://127.0.0.1:36055/ using document root <none> and password file <none>
I20260812 00:14:03.817337 6138 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:03.817387 6138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:03.817572 6138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:03.818771 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "91794131d2e6472cbc249d01e48d2b73"
format_stamp: "Formatted at 2026-08-12 00:14:03 on dist-test-slave-zpfg"
server_key: "7ce971c77a74448f505b08a7fdd90af7"
server_key_iv: "9b52ca5581e5c091a957dcef12096de0"
server_key_version: "encryptionkey@0"
I20260812 00:14:03.819152 6138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "91794131d2e6472cbc249d01e48d2b73"
format_stamp: "Formatted at 2026-08-12 00:14:03 on dist-test-slave-zpfg"
server_key: "7ce971c77a74448f505b08a7fdd90af7"
server_key_iv: "9b52ca5581e5c091a957dcef12096de0"
server_key_version: "encryptionkey@0"
I20260812 00:14:03.820616 6138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:03.821502 6154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:03.821725 6138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:03.821839 6138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "91794131d2e6472cbc249d01e48d2b73"
format_stamp: "Formatted at 2026-08-12 00:14:03 on dist-test-slave-zpfg"
server_key: "7ce971c77a74448f505b08a7fdd90af7"
server_key_iv: "9b52ca5581e5c091a957dcef12096de0"
server_key_version: "encryptionkey@0"
I20260812 00:14:03.821919 6138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:03.840646 6138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:03.843322 6138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:03.843506 6138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:03.847739 6138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38785
I20260812 00:14:03.847824 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38785 every 8 connection(s)
I20260812 00:14:03.848202 6138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:03.849306 6209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:03.852003 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73: Bootstrap starting.
I20260812 00:14:03.852622 6209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:03.852895 6209 log.cc:824] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:03.853770 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6138
I20260812 00:14:03.853828 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73: No bootstrap required, opened a new log
I20260812 00:14:03.853945 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:03.854163 16783 external_mini_cluster.cc:1428] Setting key 56c35bed505e6ea57a71228dd7f320dd
I20260812 00:14:03.855966 6209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91794131d2e6472cbc249d01e48d2b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38785 } }
I20260812 00:14:03.856161 6209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:03.856206 6209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91794131d2e6472cbc249d01e48d2b73, State: Initialized, Role: FOLLOWER
I20260812 00:14:03.856376 6209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91794131d2e6472cbc249d01e48d2b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38785 } }
I20260812 00:14:03.856467 6209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:03.856506 6209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:03.856567 6209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:03.857663 6209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91794131d2e6472cbc249d01e48d2b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38785 } }
I20260812 00:14:03.857761 6209 leader_election.cc:302] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91794131d2e6472cbc249d01e48d2b73; no voters:
I20260812 00:14:03.857972 6209 leader_election.cc:288] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:03.857995 6214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:03.858124 6214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [term 1 LEADER]: Becoming Leader. State: Replica: 91794131d2e6472cbc249d01e48d2b73, State: Running, Role: LEADER
I20260812 00:14:03.858223 6214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91794131d2e6472cbc249d01e48d2b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38785 } }
I20260812 00:14:03.858259 6209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:03.858618 6216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91794131d2e6472cbc249d01e48d2b73. Latest consensus state: current_term: 1 leader_uuid: "91794131d2e6472cbc249d01e48d2b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91794131d2e6472cbc249d01e48d2b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38785 } } }
I20260812 00:14:03.858716 6216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:03.858630 6215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91794131d2e6472cbc249d01e48d2b73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91794131d2e6472cbc249d01e48d2b73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38785 } } }
I20260812 00:14:03.858897 6215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:03.858966 6220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:03.859577 6220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:03.862712 6220 catalog_manager.cc:1347] Generated new cluster ID: 1f47ad08439b40ea8e54fc562929e4c5
I20260812 00:14:03.862778 6220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:14:03 dist-test-slave-zpfg krb5kdc[6124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:14:03.868007 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:03.854815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38941 (local address 127.0.0.1:38785)
0812 00:14:03.855033 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:03.855041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:03.855071 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:14:03.855950 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:03.855958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:03.856014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:03.856200 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:03.857041 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:03.858410 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:03.859618 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:03.860117 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:03.860284 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:03.863580 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:03.863584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:03.863585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:03.863599 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:03.866114 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:03.866595 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:03.866598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:03.866599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:03.866763 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:03.866998 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:03.866999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:03.866999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:03.867330 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:03.867488 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:03.867611 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 00:14:03.867771 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260812 00:14:03.869184 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:03.872149 6220 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:03.872670 6220 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:03.879117 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:03.870855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48975 (local address 127.0.0.1:38785)
0812 00:14:03.870898 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:03.870900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:03.871082 (+ 182us) server_negotiation.cc:407] Connection header received
0812 00:14:03.871149 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:03.871150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:03.871162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:03.871214 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:03.871981 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:03.872749 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:03.873868 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:03.874217 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:03.874291 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:03.875898 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:03.875899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:03.875900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:03.875908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:03.877681 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:03.878183 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:03.878185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:03.878185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:03.878252 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:03.878528 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:03.878529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:03.878530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:03.878673 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:03.878788 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:03.878903 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 00:14:03.879004 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:03.880836 6220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91794131d2e6472cbc249d01e48d2b73: Generated new TSK 0
I20260812 00:14:03.881018 6220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:03.886950 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6138
2026-08-12T00:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:03 dist-test-slave-zpfg krb5kdc[6240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:03 dist-test-slave-zpfg krb5kdc[6240](info): set up 2 sockets
Aug 12 00:14:03 dist-test-slave-zpfg krb5kdc[6240](info): commencing operation
krb5kdc: starting...
W20260812 00:14:04.990777 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:05 dist-test-slave-zpfg krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:05.129551 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:05.226519 6254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:05.226830 6254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:05.226882 6254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:05.228364 6254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:05.228451 6254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:05.228480 6254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:05.228525 6254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:05.228559 6254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:05.230377 6254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:05.230710 6254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:05.230981 6254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 00:14:05.236896 6254 server_base.cc:1034] running on GCE node
W20260812 00:14:05.236799 6260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:05.236855 6259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:05.236799 6262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:05.237406 6254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:05.237706 6254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:05.238864 6254 hybrid_clock.cc:648] HybridClock initialized: now 1786493645238842 us; error 41 us; skew 500 ppm
Aug 12 00:14:05 dist-test-slave-zpfg krb5kdc[6240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:05.248201 6254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:05.248816 6254 webserver.cc:460] Webserver started at http://127.0.0.1:44283/ using document root <none> and password file <none>
I20260812 00:14:05.249078 6254 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:05.249152 6254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:05.249325 6254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:05.250581 6254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "1f5f2a2b14154fe7af281983e708a7ae"
format_stamp: "Formatted at 2026-08-12 00:14:05 on dist-test-slave-zpfg"
server_key: "8966a3a42c89197023ed47ac640bf8bf"
server_key_iv: "2c4c0bfbef61ebb6b0033d56d7dd36c6"
server_key_version: "encryptionkey@0"
I20260812 00:14:05.250977 6254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "1f5f2a2b14154fe7af281983e708a7ae"
format_stamp: "Formatted at 2026-08-12 00:14:05 on dist-test-slave-zpfg"
server_key: "8966a3a42c89197023ed47ac640bf8bf"
server_key_iv: "2c4c0bfbef61ebb6b0033d56d7dd36c6"
server_key_version: "encryptionkey@0"
I20260812 00:14:05.252578 6254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:05.253683 6269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:05.253942 6254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:05.254065 6254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "1f5f2a2b14154fe7af281983e708a7ae"
format_stamp: "Formatted at 2026-08-12 00:14:05 on dist-test-slave-zpfg"
server_key: "8966a3a42c89197023ed47ac640bf8bf"
server_key_iv: "2c4c0bfbef61ebb6b0033d56d7dd36c6"
server_key_version: "encryptionkey@0"
I20260812 00:14:05.254149 6254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:05.276690 6254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:05.279299 6254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:05.279484 6254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:05.284364 6254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42681
I20260812 00:14:05.284410 6323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42681 every 8 connection(s)
I20260812 00:14:05.284811 6254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:05.285938 6324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:05.286582 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6254
I20260812 00:14:05.286717 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:05.286917 16783 external_mini_cluster.cc:1428] Setting key a34c898e06a3335a09c76d864e21d295
I20260812 00:14:05.289885 6324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae: Bootstrap starting.
I20260812 00:14:05.290789 6324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:05.291149 6324 log.cc:824] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:05.292069 6324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae: No bootstrap required, opened a new log
I20260812 00:14:05.293902 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:05.287519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50787 (local address 127.0.0.1:42681)
0812 00:14:05.287752 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:05.287764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:05.287821 (+ 57us) server_negotiation.cc:407] Connection header received
0812 00:14:05.288410 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:05.288420 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:05.288493 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:05.288669 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:05.288858 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:05.290474 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:05.291422 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:05.292067 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:05.292204 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:05.293022 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:05.293028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:05.293028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:05.293277 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:05.293324 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:05.293341 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:05.293420 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:05.293559 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 00:14:05.293640 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260812 00:14:05.294857 6324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f2a2b14154fe7af281983e708a7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260812 00:14:05.295091 6324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:05.295150 6324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5f2a2b14154fe7af281983e708a7ae, State: Initialized, Role: FOLLOWER
I20260812 00:14:05.295298 6324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f2a2b14154fe7af281983e708a7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260812 00:14:05.295408 6324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:05.295459 6324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:05.295514 6324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:05.295620 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:05.296265 6324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f2a2b14154fe7af281983e708a7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260812 00:14:05.296368 6324 leader_election.cc:302] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f5f2a2b14154fe7af281983e708a7ae; no voters:
I20260812 00:14:05.296550 6324 leader_election.cc:288] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:05.296599 6329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:05.296787 6329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [term 1 LEADER]: Becoming Leader. State: Replica: 1f5f2a2b14154fe7af281983e708a7ae, State: Running, Role: LEADER
I20260812 00:14:05.296865 6324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:05.296903 6329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f2a2b14154fe7af281983e708a7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } }
I20260812 00:14:05.297322 6329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f5f2a2b14154fe7af281983e708a7ae. Latest consensus state: current_term: 1 leader_uuid: "1f5f2a2b14154fe7af281983e708a7ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f2a2b14154fe7af281983e708a7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } } }
I20260812 00:14:05.297333 6332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f5f2a2b14154fe7af281983e708a7ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f2a2b14154fe7af281983e708a7ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42681 } } }
I20260812 00:14:05.297572 6332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:05.297785 6329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:05.297870 6346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:05.298445 6346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:05.300813 6346 catalog_manager.cc:1347] Generated new cluster ID: 6e23c7ab10054316bed6ab1f4f478eef
I20260812 00:14:05.300925 6346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:05.300993 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:05.297190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51845 (local address 127.0.0.1:42681)
0812 00:14:05.297233 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:05.297235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:05.297350 (+ 115us) server_negotiation.cc:407] Connection header received
0812 00:14:05.297462 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:05.297463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:05.297473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:05.297503 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:05.297745 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:05.298500 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:05.299447 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:05.299762 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:05.299845 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:05.300578 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:05.300580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:05.300580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:05.300586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:05.300602 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:05.300606 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:05.300654 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:05.300842 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 00:14:05.300873 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:05.315445 6346 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:05.316318 6346 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:05.321383 6346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f5f2a2b14154fe7af281983e708a7ae: Generated new TSK 0
I20260812 00:14:05.321616 6346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:05.326958 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6254
2026-08-12T00:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:05 dist-test-slave-zpfg krb5kdc[6358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:05 dist-test-slave-zpfg krb5kdc[6358](info): set up 2 sockets
Aug 12 00:14:05 dist-test-slave-zpfg krb5kdc[6358](info): commencing operation
krb5kdc: starting...
W20260812 00:14:06.462698 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:06 dist-test-slave-zpfg krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:06.598186 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:06.692571 6372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:06.692870 6372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:06.692915 6372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:06.694417 6372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:06.694510 6372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:06.694535 6372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:06.694566 6372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:06.694597 6372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:06.696597 6372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:06.696967 6372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:06.697253 6372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:06.703013 6380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:06.703024 6378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:06.703023 6377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:06.703550 6372 server_base.cc:1034] running on GCE node
I20260812 00:14:06.703860 6372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:06.704162 6372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:06.705330 6372 hybrid_clock.cc:648] HybridClock initialized: now 1786493646705304 us; error 43 us; skew 500 ppm
Aug 12 00:14:06 dist-test-slave-zpfg krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:06.708377 6372 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:06.708988 6372 webserver.cc:460] Webserver started at http://127.0.0.1:37295/ using document root <none> and password file <none>
I20260812 00:14:06.709266 6372 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:06.709319 6372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:06.709425 6372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:06.710759 6372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "b483bfe046a64d819dbe13ab4feae7f1"
format_stamp: "Formatted at 2026-08-12 00:14:06 on dist-test-slave-zpfg"
server_key: "17a75bf1e009885cba2720e007851884"
server_key_iv: "f4c06b5b11ecaa14df369f019896765c"
server_key_version: "encryptionkey@0"
I20260812 00:14:06.711118 6372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "b483bfe046a64d819dbe13ab4feae7f1"
format_stamp: "Formatted at 2026-08-12 00:14:06 on dist-test-slave-zpfg"
server_key: "17a75bf1e009885cba2720e007851884"
server_key_iv: "f4c06b5b11ecaa14df369f019896765c"
server_key_version: "encryptionkey@0"
I20260812 00:14:06.712762 6372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 00:14:06.713691 6387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:06.713919 6372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:06.714021 6372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "b483bfe046a64d819dbe13ab4feae7f1"
format_stamp: "Formatted at 2026-08-12 00:14:06 on dist-test-slave-zpfg"
server_key: "17a75bf1e009885cba2720e007851884"
server_key_iv: "f4c06b5b11ecaa14df369f019896765c"
server_key_version: "encryptionkey@0"
I20260812 00:14:06.714082 6372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:06.717561 6372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:06.720363 6372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:06.720503 6372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:06.724973 6372 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39077
I20260812 00:14:06.725070 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39077 every 8 connection(s)
I20260812 00:14:06.725445 6372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:06.726552 6442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:06.729029 6442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1: Bootstrap starting.
I20260812 00:14:06.729700 6442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:06.730014 6442 log.cc:824] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:06.730923 6442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1: No bootstrap required, opened a new log
I20260812 00:14:06.732659 6442 raft_consensus.cc:348] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b483bfe046a64d819dbe13ab4feae7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39077 } }
I20260812 00:14:06.732846 6442 raft_consensus.cc:374] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:06.732896 6442 raft_consensus.cc:729] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b483bfe046a64d819dbe13ab4feae7f1, State: Initialized, Role: FOLLOWER
I20260812 00:14:06.733074 6442 consensus_queue.cc:260] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b483bfe046a64d819dbe13ab4feae7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39077 } }
I20260812 00:14:06.733175 6442 raft_consensus.cc:388] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:06.733222 6442 raft_consensus.cc:482] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:06.733281 6442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:06.733862 6442 raft_consensus.cc:504] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b483bfe046a64d819dbe13ab4feae7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39077 } }
I20260812 00:14:06.733990 6442 leader_election.cc:302] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b483bfe046a64d819dbe13ab4feae7f1; no voters:
I20260812 00:14:06.734200 6442 leader_election.cc:288] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:06.734413 6445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:06.734511 6442 sys_catalog.cc:564] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:06.734598 6445 raft_consensus.cc:686] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [term 1 LEADER]: Becoming Leader. State: Replica: b483bfe046a64d819dbe13ab4feae7f1, State: Running, Role: LEADER
I20260812 00:14:06.734748 6445 consensus_queue.cc:237] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b483bfe046a64d819dbe13ab4feae7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39077 } }
I20260812 00:14:06.735061 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6372
I20260812 00:14:06.735193 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:06.735420 16783 external_mini_cluster.cc:1428] Setting key 3d8d71dbca23a276900d0aca2daf32ae
I20260812 00:14:06.735569 6447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b483bfe046a64d819dbe13ab4feae7f1. Latest consensus state: current_term: 1 leader_uuid: "b483bfe046a64d819dbe13ab4feae7f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b483bfe046a64d819dbe13ab4feae7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39077 } } }
I20260812 00:14:06.735770 6447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:06.735790 6446 sys_catalog.cc:455] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b483bfe046a64d819dbe13ab4feae7f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b483bfe046a64d819dbe13ab4feae7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39077 } } }
I20260812 00:14:06.735849 6446 sys_catalog.cc:458] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:06.736158 6459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:06.736809 6459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:06.738427 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:06.736173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52255 (local address 127.0.0.1:39077)
0812 00:14:06.736347 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:06.736358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:06.736813 (+ 455us) server_negotiation.cc:407] Connection header received
0812 00:14:06.736957 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:06.736965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:06.737002 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:06.737147 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:06.737862 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:06.737866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:06.737867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:06.737872 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:06.737914 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:06.737928 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:06.737972 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:06.738142 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 00:14:06.738189 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260812 00:14:06.738581 6459 catalog_manager.cc:1347] Generated new cluster ID: b9349b42500048d498602f4b5728aa4f
I20260812 00:14:06.738639 6459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:06.743337 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:06.746206 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:06.744950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35095 (local address 127.0.0.1:39077)
0812 00:14:06.744993 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:06.744995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:06.745131 (+ 136us) server_negotiation.cc:407] Connection header received
0812 00:14:06.745208 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:06.745209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:06.745215 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:06.745267 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:06.745963 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:06.745964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:06.745965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:06.745970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:06.745987 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:06.745992 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:06.746044 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:06.746099 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 00:14:06.746116 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:06.747885 6459 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:06.748077 6459 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:06.752651 6459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b483bfe046a64d819dbe13ab4feae7f1: Generated new TSK 0
I20260812 00:14:06.752877 6459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:06.758322 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6372
2026-08-12T00:14:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:06 dist-test-slave-zpfg krb5kdc[6473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:06 dist-test-slave-zpfg krb5kdc[6473](info): set up 2 sockets
Aug 12 00:14:06 dist-test-slave-zpfg krb5kdc[6473](info): commencing operation
krb5kdc: starting...
W20260812 00:14:07.846529 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:07 dist-test-slave-zpfg krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:07.981606 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:08.072679 6487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:08.072959 6487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:08.073004 6487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:08.074249 6487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:08.074333 6487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:08.074385 6487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:08.074420 6487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:08.074450 6487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:08.076037 6487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:08.076364 6487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:08.076607 6487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:08.081881 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:08.081874 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:08.081930 6487 server_base.cc:1034] running on GCE node
W20260812 00:14:08.081908 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:08.082326 6487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:08.082688 6487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:08.083865 6487 hybrid_clock.cc:648] HybridClock initialized: now 1786493648083848 us; error 29 us; skew 500 ppm
Aug 12 00:14:08 dist-test-slave-zpfg krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:08.086597 6487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:08.087021 6487 webserver.cc:460] Webserver started at http://127.0.0.1:42667/ using document root <none> and password file <none>
I20260812 00:14:08.087236 6487 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:08.087278 6487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:08.087378 6487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:08.088488 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "ff1003a0f1924f90a9fb56e5d7b5071d"
format_stamp: "Formatted at 2026-08-12 00:14:08 on dist-test-slave-zpfg"
server_key: "fcaf2f4429a97c52f4e42edaadbfd966"
server_key_iv: "f96f894e058fc98025ccbb5435635ce9"
server_key_version: "encryptionkey@0"
I20260812 00:14:08.088815 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "ff1003a0f1924f90a9fb56e5d7b5071d"
format_stamp: "Formatted at 2026-08-12 00:14:08 on dist-test-slave-zpfg"
server_key: "fcaf2f4429a97c52f4e42edaadbfd966"
server_key_iv: "f96f894e058fc98025ccbb5435635ce9"
server_key_version: "encryptionkey@0"
I20260812 00:14:08.090428 6487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260812 00:14:08.091522 6504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:08.091710 6487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:08.091845 6487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "ff1003a0f1924f90a9fb56e5d7b5071d"
format_stamp: "Formatted at 2026-08-12 00:14:08 on dist-test-slave-zpfg"
server_key: "fcaf2f4429a97c52f4e42edaadbfd966"
server_key_iv: "f96f894e058fc98025ccbb5435635ce9"
server_key_version: "encryptionkey@0"
I20260812 00:14:08.091909 6487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:08.121928 6487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:08.124338 6487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:08.124469 6487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:08.129099 6487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33825
I20260812 00:14:08.129141 6558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33825 every 8 connection(s)
I20260812 00:14:08.129525 6487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:08.130643 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:08.133256 6559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d: Bootstrap starting.
I20260812 00:14:08.133900 6559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:08.134167 6559 log.cc:824] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:08.134994 6559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d: No bootstrap required, opened a new log
I20260812 00:14:08.136909 6559 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260812 00:14:08.137077 6559 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:08.137102 6559 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1003a0f1924f90a9fb56e5d7b5071d, State: Initialized, Role: FOLLOWER
I20260812 00:14:08.137280 6559 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260812 00:14:08.137377 6559 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:08.137403 6559 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:08.137435 6559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:08.138019 6559 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260812 00:14:08.138136 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6487
I20260812 00:14:08.138120 6559 leader_election.cc:302] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff1003a0f1924f90a9fb56e5d7b5071d; no voters:
I20260812 00:14:08.138288 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:08.138432 6559 leader_election.cc:288] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:08.138466 6562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:08.138536 16783 external_mini_cluster.cc:1428] Setting key d685056e03835678dece04f08795f34c
I20260812 00:14:08.138599 6562 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [term 1 LEADER]: Becoming Leader. State: Replica: ff1003a0f1924f90a9fb56e5d7b5071d, State: Running, Role: LEADER
I20260812 00:14:08.138695 6562 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } }
I20260812 00:14:08.138743 6559 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:08.139520 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } } }
I20260812 00:14:08.139760 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:08.139669 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1003a0f1924f90a9fb56e5d7b5071d. Latest consensus state: current_term: 1 leader_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1003a0f1924f90a9fb56e5d7b5071d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33825 } } }
I20260812 00:14:08.140050 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:08.140234 6574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:08.140861 6574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:08.143002 6574 catalog_manager.cc:1347] Generated new cluster ID: 2da88eeb1c094ba780a8b019d6cb6628
I20260812 00:14:08.143065 6574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 00:14:08 dist-test-slave-zpfg krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:14:08.152062 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:08.139170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:57771 (local address 127.0.0.1:33825)
0812 00:14:08.139380 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:08.139391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:08.139980 (+ 589us) server_negotiation.cc:407] Connection header received
0812 00:14:08.140254 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:08.140265 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:08.140345 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:08.140506 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:08.141552 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:08.142687 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:08.143837 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:08.144470 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:08.144558 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:08.147584 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:08.147589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:08.147589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:08.147602 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:08.149784 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:08.150255 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:08.150258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:08.150259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:08.150408 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:08.150655 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:08.150656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:08.150656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:08.150971 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:08.151137 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:08.151701 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 00:14:08.151812 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260812 00:14:08.152654 6574 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:08.153159 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:08.153374 6574 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:08.162401 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:08.154735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:55585 (local address 127.0.0.1:33825)
0812 00:14:08.154779 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:08.154782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:08.154906 (+ 124us) server_negotiation.cc:407] Connection header received
0812 00:14:08.154972 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:08.154973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:08.154984 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:08.155035 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:08.155576 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:08.156154 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:08.157188 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:08.157485 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:08.157534 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:08.159459 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:08.159461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:08.159461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:08.159469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:08.161133 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:08.161535 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:08.161536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:08.161537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:08.161600 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:08.161842 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:08.161843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:08.161843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:08.161967 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:08.162091 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:08.162200 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 00:14:08.162263 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:14:08.165257 6574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff1003a0f1924f90a9fb56e5d7b5071d: Generated new TSK 0
I20260812 00:14:08.165473 6574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:08.174038 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6487
W20260812 00:14:08.180490 6471 connection.cc:535] client connection to 127.0.0.1:33825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33825): Connection reset by peer (error 104)
2026-08-12T00:14:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:08 dist-test-slave-zpfg krb5kdc[6590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:08 dist-test-slave-zpfg krb5kdc[6590](info): set up 2 sockets
Aug 12 00:14:08 dist-test-slave-zpfg krb5kdc[6590](info): commencing operation
krb5kdc: starting...
W20260812 00:14:09.274009 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:09 dist-test-slave-zpfg krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:09.414558 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:09.499130 6603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:09.499419 6603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:09.499466 6603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:09.500720 6603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:09.500800 6603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:09.500825 6603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:09.500849 6603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:09.500877 6603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:09.502472 6603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:09.502768 6603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:09.503015 6603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:09.508442 6611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:09.508436 6609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:09.508440 6608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:09.508683 6603 server_base.cc:1034] running on GCE node
I20260812 00:14:09.509084 6603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:09.509433 6603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:09.510604 6603 hybrid_clock.cc:648] HybridClock initialized: now 1786493649510582 us; error 55 us; skew 500 ppm
Aug 12 00:14:09 dist-test-slave-zpfg krb5kdc[6590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:09.513226 6603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:09.513794 6603 webserver.cc:460] Webserver started at http://127.0.0.1:38795/ using document root <none> and password file <none>
I20260812 00:14:09.514063 6603 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:09.514135 6603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:09.514297 6603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:09.515427 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "e86f35515e40457d80fb685f84565cf3"
format_stamp: "Formatted at 2026-08-12 00:14:09 on dist-test-slave-zpfg"
server_key: "8104c690cd76b2db5f62e030609caf40"
server_key_iv: "8731226af73182ea6414afee5b42afc1"
server_key_version: "encryptionkey@0"
I20260812 00:14:09.515825 6603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "e86f35515e40457d80fb685f84565cf3"
format_stamp: "Formatted at 2026-08-12 00:14:09 on dist-test-slave-zpfg"
server_key: "8104c690cd76b2db5f62e030609caf40"
server_key_iv: "8731226af73182ea6414afee5b42afc1"
server_key_version: "encryptionkey@0"
I20260812 00:14:09.517298 6603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:09.518057 6619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:09.518302 6603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:09.518442 6603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "e86f35515e40457d80fb685f84565cf3"
format_stamp: "Formatted at 2026-08-12 00:14:09 on dist-test-slave-zpfg"
server_key: "8104c690cd76b2db5f62e030609caf40"
server_key_iv: "8731226af73182ea6414afee5b42afc1"
server_key_version: "encryptionkey@0"
I20260812 00:14:09.518524 6603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:09.546973 6603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:09.549301 6603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:09.549512 6603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:09.553853 6603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46723
I20260812 00:14:09.553897 6673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46723 every 8 connection(s)
I20260812 00:14:09.554296 6603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:09.555356 6674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:09.557930 6674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3: Bootstrap starting.
I20260812 00:14:09.558631 6674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:09.558935 6674 log.cc:824] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:09.559783 6674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3: No bootstrap required, opened a new log
I20260812 00:14:09.561050 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6603
I20260812 00:14:09.561192 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:09.561398 16783 external_mini_cluster.cc:1428] Setting key ab2eecbae75c98f17548ca1a4ab6856a
I20260812 00:14:09.561890 6674 raft_consensus.cc:348] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86f35515e40457d80fb685f84565cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260812 00:14:09.562144 6674 raft_consensus.cc:374] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:09.562230 6674 raft_consensus.cc:729] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86f35515e40457d80fb685f84565cf3, State: Initialized, Role: FOLLOWER
I20260812 00:14:09.562412 6674 consensus_queue.cc:260] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86f35515e40457d80fb685f84565cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260812 00:14:09.562496 6674 raft_consensus.cc:388] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:09.562529 6674 raft_consensus.cc:482] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:09.562587 6674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:09.563841 6674 raft_consensus.cc:504] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86f35515e40457d80fb685f84565cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260812 00:14:09.563956 6674 leader_election.cc:302] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e86f35515e40457d80fb685f84565cf3; no voters:
I20260812 00:14:09.564110 6674 leader_election.cc:288] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:09.564174 6679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:09.564352 6674 sys_catalog.cc:564] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:09.564417 6679 raft_consensus.cc:686] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [term 1 LEADER]: Becoming Leader. State: Replica: e86f35515e40457d80fb685f84565cf3, State: Running, Role: LEADER
I20260812 00:14:09.564549 6679 consensus_queue.cc:237] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86f35515e40457d80fb685f84565cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260812 00:14:09.565902 6681 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e86f35515e40457d80fb685f84565cf3. Latest consensus state: current_term: 1 leader_uuid: "e86f35515e40457d80fb685f84565cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86f35515e40457d80fb685f84565cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } } }
I20260812 00:14:09.565912 6680 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e86f35515e40457d80fb685f84565cf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86f35515e40457d80fb685f84565cf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } } }
I20260812 00:14:09.565994 6681 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:09.566022 6680 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3 [sys.catalog]: This master's current role is: LEADER
W20260812 00:14:09.566653 6695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:09.566708 6695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:09.566756 6696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:09.566934 6696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:09.567981 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:09.562026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:56443 (local address 127.0.0.1:46723)
0812 00:14:09.562282 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:09.562293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:09.562325 (+ 32us) server_negotiation.cc:407] Connection header received
0812 00:14:09.562470 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:09.562478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:09.562545 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:09.562700 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:09.562966 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:09.564453 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:09.565537 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:09.566218 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:09.566326 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:09.567206 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:09.567210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:09.567210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:09.567422 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:09.567462 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:09.567477 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:09.567561 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:09.567720 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 00:14:09.567759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":129,"threads_started":1}
I20260812 00:14:09.568598 6696 catalog_manager.cc:1347] Generated new cluster ID: a8334817c4a64cd39bb4dc7bbfabda34
I20260812 00:14:09.568652 6696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:09.569058 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:09.574461 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:09.570673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:56987 (local address 127.0.0.1:46723)
0812 00:14:09.570718 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:09.570720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:09.570941 (+ 221us) server_negotiation.cc:407] Connection header received
0812 00:14:09.570988 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:09.570989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:09.571002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:09.571049 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:09.571183 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:09.571929 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:09.572950 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:09.573271 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:09.573355 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:09.574147 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:09.574149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:09.574149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:09.574154 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:09.574172 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:09.574176 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:09.574237 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:09.574334 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 00:14:09.574374 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:14:09.588936 6696 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:09.589732 6696 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:09.605247 6696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e86f35515e40457d80fb685f84565cf3: Generated new TSK 0
I20260812 00:14:09.605508 6696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:09.612499 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6603
W20260812 00:14:09.618433 6588 connection.cc:535] client connection to 127.0.0.1:46723 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46723): Connection reset by peer (error 104)
2026-08-12T00:14:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:09 dist-test-slave-zpfg krb5kdc[6706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:09 dist-test-slave-zpfg krb5kdc[6706](info): set up 2 sockets
Aug 12 00:14:09 dist-test-slave-zpfg krb5kdc[6706](info): commencing operation
krb5kdc: starting...
W20260812 00:14:10.766991 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:10 dist-test-slave-zpfg krb5kdc[6706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:10.907727 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:10.995520 6719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:10.995770 6719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:10.995796 6719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:10.997108 6719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:10.997174 6719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:10.997197 6719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:10.997210 6719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:10.997222 6719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:10.998868 6719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:10.999123 6719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:10.999311 6719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:11.005165 6726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:11.005177 6727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:11.005564 6729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:11.006915 6719 server_base.cc:1034] running on GCE node
I20260812 00:14:11.007442 6719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:11.007862 6719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:11.009049 6719 hybrid_clock.cc:648] HybridClock initialized: now 1786493651009023 us; error 43 us; skew 500 ppm
Aug 12 00:14:11 dist-test-slave-zpfg krb5kdc[6706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:11.017084 6719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:11.017649 6719 webserver.cc:460] Webserver started at http://127.0.0.1:44723/ using document root <none> and password file <none>
I20260812 00:14:11.017971 6719 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:11.018056 6719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:11.018232 6719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:11.019569 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "b78b1c5caf184327a8edb6660d8cbbd2"
format_stamp: "Formatted at 2026-08-12 00:14:11 on dist-test-slave-zpfg"
server_key: "394e4e37e36f79f98f2f2e3a09dd9ec8"
server_key_iv: "a0351d52046c2a4f651f136252912cbe"
server_key_version: "encryptionkey@0"
I20260812 00:14:11.020005 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "b78b1c5caf184327a8edb6660d8cbbd2"
format_stamp: "Formatted at 2026-08-12 00:14:11 on dist-test-slave-zpfg"
server_key: "394e4e37e36f79f98f2f2e3a09dd9ec8"
server_key_iv: "a0351d52046c2a4f651f136252912cbe"
server_key_version: "encryptionkey@0"
I20260812 00:14:11.021587 6719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:11.022769 6737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:11.023010 6719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:11.023138 6719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "b78b1c5caf184327a8edb6660d8cbbd2"
format_stamp: "Formatted at 2026-08-12 00:14:11 on dist-test-slave-zpfg"
server_key: "394e4e37e36f79f98f2f2e3a09dd9ec8"
server_key_iv: "a0351d52046c2a4f651f136252912cbe"
server_key_version: "encryptionkey@0"
I20260812 00:14:11.023229 6719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:11.027269 6719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:11.029700 6719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:11.029891 6719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:11.034868 6719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37253
I20260812 00:14:11.034906 6798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37253 every 8 connection(s)
I20260812 00:14:11.035312 6719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:11.036270 6799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:11.038877 6799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2: Bootstrap starting.
I20260812 00:14:11.039502 6799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:11.039819 6799 log.cc:824] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:11.040627 6799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2: No bootstrap required, opened a new log
I20260812 00:14:11.042574 6799 raft_consensus.cc:348] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37253 } }
I20260812 00:14:11.042737 6799 raft_consensus.cc:374] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:11.042761 6799 raft_consensus.cc:729] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78b1c5caf184327a8edb6660d8cbbd2, State: Initialized, Role: FOLLOWER
I20260812 00:14:11.042932 6799 consensus_queue.cc:260] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37253 } }
I20260812 00:14:11.042997 6799 raft_consensus.cc:388] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:11.043020 6799 raft_consensus.cc:482] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:11.043071 6799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:11.043668 6799 raft_consensus.cc:504] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37253 } }
I20260812 00:14:11.043766 6799 leader_election.cc:302] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b78b1c5caf184327a8edb6660d8cbbd2; no voters:
I20260812 00:14:11.043989 6799 leader_election.cc:288] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:11.044068 6802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:11.044267 6799 sys_catalog.cc:564] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:11.044351 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6719
I20260812 00:14:11.044494 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:11.044849 16783 external_mini_cluster.cc:1428] Setting key 1364641dc94553d3a505041023f7b4e2
I20260812 00:14:11.045442 6802 raft_consensus.cc:686] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [term 1 LEADER]: Becoming Leader. State: Replica: b78b1c5caf184327a8edb6660d8cbbd2, State: Running, Role: LEADER
I20260812 00:14:11.045698 6802 consensus_queue.cc:237] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37253 } }
I20260812 00:14:11.046237 6810 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b78b1c5caf184327a8edb6660d8cbbd2. Latest consensus state: current_term: 1 leader_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37253 } } }
I20260812 00:14:11.046422 6810 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:11.046427 6803 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78b1c5caf184327a8edb6660d8cbbd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37253 } } }
I20260812 00:14:11.046564 6803 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:11.046795 6821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:11.047746 6821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:11.047770 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:11.045640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:46191 (local address 127.0.0.1:37253)
0812 00:14:11.045827 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:11.045838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:11.045886 (+ 48us) server_negotiation.cc:407] Connection header received
0812 00:14:11.046011 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:11.046019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:11.046053 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:11.046757 (+ 704us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:11.047160 (+ 403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:11.047164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:11.047165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:11.047169 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:11.047300 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:11.047321 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:11.047396 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:11.047492 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 00:14:11.047547 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":56,"threads_started":1}
I20260812 00:14:11.048789 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:11.049408 6821 catalog_manager.cc:1347] Generated new cluster ID: a37b03ab8c0f40fdb1f4f0662b76e78c
I20260812 00:14:11.049467 6821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:11.051637 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:11.050406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:57155 (local address 127.0.0.1:37253)
0812 00:14:11.050444 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:11.050446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:11.050683 (+ 237us) server_negotiation.cc:407] Connection header received
0812 00:14:11.050714 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:11.050715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:11.050721 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:11.050790 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:11.051413 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:11.051415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:11.051415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:11.051419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:11.051438 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:11.051442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:11.051506 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:11.051556 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 00:14:11.051570 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:14:11.071316 6821 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:11.071538 6821 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:11.080808 6821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b78b1c5caf184327a8edb6660d8cbbd2: Generated new TSK 0
I20260812 00:14:11.081063 6821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:11.088424 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6719
2026-08-12T00:14:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:11 dist-test-slave-zpfg krb5kdc[6832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:11 dist-test-slave-zpfg krb5kdc[6832](info): set up 2 sockets
Aug 12 00:14:11 dist-test-slave-zpfg krb5kdc[6832](info): commencing operation
krb5kdc: starting...
W20260812 00:14:12.203745 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:12 dist-test-slave-zpfg krb5kdc[6832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:12.341897 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:12.445065 6848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:12.445351 6848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:12.445396 6848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:12.446647 6848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:12.446728 6848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:12.446753 6848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:12.446774 6848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:12.446806 6848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:12.448365 6848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:12.448668 6848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:12.448912 6848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:12.454597 6857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:12.454727 6848 server_base.cc:1034] running on GCE node
W20260812 00:14:12.454597 6855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:12.454704 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:12.455211 6848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:12.455536 6848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:12.456753 6848 hybrid_clock.cc:648] HybridClock initialized: now 1786493652456722 us; error 38 us; skew 500 ppm
Aug 12 00:14:12 dist-test-slave-zpfg krb5kdc[6832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:12.459937 6848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:12.460606 6848 webserver.cc:460] Webserver started at http://127.0.0.1:40339/ using document root <none> and password file <none>
I20260812 00:14:12.460920 6848 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:12.461026 6848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:12.461323 6848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:12.462652 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "e1135e54b88c45cd8c345af2779e9fa9"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zpfg"
server_key: "a0bf43e1a8b7290fc489eb22a712f247"
server_key_iv: "3f273e82ba680163690328edc5826db5"
server_key_version: "encryptionkey@0"
I20260812 00:14:12.463054 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "e1135e54b88c45cd8c345af2779e9fa9"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zpfg"
server_key: "a0bf43e1a8b7290fc489eb22a712f247"
server_key_iv: "3f273e82ba680163690328edc5826db5"
server_key_version: "encryptionkey@0"
I20260812 00:14:12.464615 6848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:12.465556 6864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:12.465925 6848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:12.466147 6848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "e1135e54b88c45cd8c345af2779e9fa9"
format_stamp: "Formatted at 2026-08-12 00:14:12 on dist-test-slave-zpfg"
server_key: "a0bf43e1a8b7290fc489eb22a712f247"
server_key_iv: "3f273e82ba680163690328edc5826db5"
server_key_version: "encryptionkey@0"
I20260812 00:14:12.466254 6848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:12.477325 6848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:12.480327 6848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:12.480525 6848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:12.485653 6848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36373
I20260812 00:14:12.485730 6921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36373 every 8 connection(s)
I20260812 00:14:12.486146 6848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:12.487151 6922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:12.488282 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6848
I20260812 00:14:12.488411 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:12.488615 16783 external_mini_cluster.cc:1428] Setting key 8a9569cb829d0325eea3c1088d38d86d
I20260812 00:14:12.491036 6922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9: Bootstrap starting.
I20260812 00:14:12.491667 6922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:12.491984 6922 log.cc:824] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:12.492690 6922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9: No bootstrap required, opened a new log
I20260812 00:14:12.494720 6922 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1135e54b88c45cd8c345af2779e9fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36373 } }
I20260812 00:14:12.494860 6922 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:12.494884 6922 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1135e54b88c45cd8c345af2779e9fa9, State: Initialized, Role: FOLLOWER
I20260812 00:14:12.495025 6922 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1135e54b88c45cd8c345af2779e9fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36373 } }
I20260812 00:14:12.495103 6922 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:12.495126 6922 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:12.495189 6922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:12.495836 6922 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1135e54b88c45cd8c345af2779e9fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36373 } }
I20260812 00:14:12.495965 6922 leader_election.cc:302] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1135e54b88c45cd8c345af2779e9fa9; no voters:
I20260812 00:14:12.496194 6922 leader_election.cc:288] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:12.496223 6927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:12.496349 6927 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [term 1 LEADER]: Becoming Leader. State: Replica: e1135e54b88c45cd8c345af2779e9fa9, State: Running, Role: LEADER
Aug 12 00:14:12 dist-test-slave-zpfg krb5kdc[6832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 00:14:12.496501 6922 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:12.496554 6927 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1135e54b88c45cd8c345af2779e9fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36373 } }
I20260812 00:14:12.496927 6929 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1135e54b88c45cd8c345af2779e9fa9. Latest consensus state: current_term: 1 leader_uuid: "e1135e54b88c45cd8c345af2779e9fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1135e54b88c45cd8c345af2779e9fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36373 } } }
I20260812 00:14:12.497014 6929 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:12.497256 6928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1135e54b88c45cd8c345af2779e9fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1135e54b88c45cd8c345af2779e9fa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36373 } } }
I20260812 00:14:12.497401 6928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:12.497726 6934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:12.498404 6934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:12.500344 6934 catalog_manager.cc:1347] Generated new cluster ID: cb12305caf9840b892eed1489ed7cc2a
I20260812 00:14:12.500402 6934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:12.501521 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.489232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:60221 (local address 127.0.0.1:36373)
0812 00:14:12.489431 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:12.489441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:12.489478 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:14:12.489910 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:12.489917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:12.489968 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:12.490093 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:12.490969 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.492284 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.493232 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.493718 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.493850 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.497234 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:12.497239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:12.497239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:12.497255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:12.499585 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.500158 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.500162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.500162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.500281 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.500527 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.500528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.500528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.500859 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:12.501013 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:12.501193 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 00:14:12.501299 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260812 00:14:12.502823 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:12.512755 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:12.504608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:49851 (local address 127.0.0.1:36373)
0812 00:14:12.504688 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:12.504690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:12.504699 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:14:12.504728 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:12.504729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:12.504739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:12.504806 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:12.505533 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.506320 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.507425 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:12.507797 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:12.507841 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:12.509513 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:12.509515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:12.509515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:12.509523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:12.511379 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.511865 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.511867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.511867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.511933 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:12.512175 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:12.512176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:12.512177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:12.512322 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:12.512430 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:12.512575 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 00:14:12.512644 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 00:14:12.514900 6934 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:12.515618 6934 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:12.518945 6934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1135e54b88c45cd8c345af2779e9fa9: Generated new TSK 0
I20260812 00:14:12.519145 6934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:12.522832 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6848
W20260812 00:14:12.529630 6830 connection.cc:535] client connection to 127.0.0.1:36373 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36373): Connection reset by peer (error 104)
2026-08-12T00:14:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:12 dist-test-slave-zpfg krb5kdc[6956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:12 dist-test-slave-zpfg krb5kdc[6956](info): set up 2 sockets
Aug 12 00:14:12 dist-test-slave-zpfg krb5kdc[6956](info): commencing operation
krb5kdc: starting...
W20260812 00:14:13.633005 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:13 dist-test-slave-zpfg krb5kdc[6956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:13.765616 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:13.855376 6969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:13.855659 6969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:13.855705 6969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:13.857093 6969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:13.857180 6969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:13.857218 6969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:13.857254 6969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:13.857282 6969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:13.859848 6969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:13.860214 6969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:13.860492 6969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:13.866158 6977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:13.866170 6975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:13.866257 6969 server_base.cc:1034] running on GCE node
W20260812 00:14:13.866257 6974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:13.866784 6969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:13.867063 6969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:13.868224 6969 hybrid_clock.cc:648] HybridClock initialized: now 1786493653868242 us; error 66 us; skew 500 ppm
Aug 12 00:14:13 dist-test-slave-zpfg krb5kdc[6956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:13.871050 6969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:13.871541 6969 webserver.cc:460] Webserver started at http://127.0.0.1:33705/ using document root <none> and password file <none>
I20260812 00:14:13.871779 6969 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:13.871826 6969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:13.871927 6969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:13.873103 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "4a45a19c99d9448cafbdf3855ad1a3ac"
format_stamp: "Formatted at 2026-08-12 00:14:13 on dist-test-slave-zpfg"
server_key: "90b1324c572bfcb265271d4f01d3726c"
server_key_iv: "ce18c3b598211907b41ffaeb708bbb40"
server_key_version: "encryptionkey@0"
I20260812 00:14:13.873433 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "4a45a19c99d9448cafbdf3855ad1a3ac"
format_stamp: "Formatted at 2026-08-12 00:14:13 on dist-test-slave-zpfg"
server_key: "90b1324c572bfcb265271d4f01d3726c"
server_key_iv: "ce18c3b598211907b41ffaeb708bbb40"
server_key_version: "encryptionkey@0"
I20260812 00:14:13.875088 6969 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 00:14:13.876013 6984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:13.876492 6969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:13.876665 6969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "4a45a19c99d9448cafbdf3855ad1a3ac"
format_stamp: "Formatted at 2026-08-12 00:14:13 on dist-test-slave-zpfg"
server_key: "90b1324c572bfcb265271d4f01d3726c"
server_key_iv: "ce18c3b598211907b41ffaeb708bbb40"
server_key_version: "encryptionkey@0"
I20260812 00:14:13.876782 6969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:13.892701 6969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:13.896291 6969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:13.896494 6969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:13.901347 6969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33249
I20260812 00:14:13.901444 7042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33249 every 8 connection(s)
I20260812 00:14:13.901876 6969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:13.902172 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 6969
I20260812 00:14:13.902388 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:13.902616 16783 external_mini_cluster.cc:1428] Setting key ba9b18667d01d6984f0d37652bf95846
I20260812 00:14:13.903311 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:13.906802 7043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac: Bootstrap starting.
I20260812 00:14:13.907662 7043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:13.908087 7043 log.cc:824] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:13.909018 7043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac: No bootstrap required, opened a new log
I20260812 00:14:13.909906 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.903244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:41499 (local address 127.0.0.1:33249)
0812 00:14:13.903458 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:13.903467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:13.903497 (+ 30us) server_negotiation.cc:407] Connection header received
0812 00:14:13.903971 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:13.903979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:13.904039 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:13.904178 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:13.904486 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.905691 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.907034 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.907779 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.908006 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.909021 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:13.909026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:13.909027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:13.909272 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:13.909333 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:13.909352 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:13.909453 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:13.909597 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 00:14:13.909632 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260812 00:14:13.911360 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:13.911993 7043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260812 00:14:13.912221 7043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:13.912277 7043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a45a19c99d9448cafbdf3855ad1a3ac, State: Initialized, Role: FOLLOWER
I20260812 00:14:13.912432 7043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260812 00:14:13.912528 7043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:13.912569 7043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:13.912626 7043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:13.913646 7043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260812 00:14:13.913779 7043 leader_election.cc:302] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a45a19c99d9448cafbdf3855ad1a3ac; no voters:
I20260812 00:14:13.914006 7043 leader_election.cc:288] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:13.914040 7053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:13.914170 7053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [term 1 LEADER]: Becoming Leader. State: Replica: 4a45a19c99d9448cafbdf3855ad1a3ac, State: Running, Role: LEADER
I20260812 00:14:13.914289 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } }
I20260812 00:14:13.914324 7043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:13.914714 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } } }
I20260812 00:14:13.914866 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:13.915122 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a45a19c99d9448cafbdf3855ad1a3ac. Latest consensus state: current_term: 1 leader_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a45a19c99d9448cafbdf3855ad1a3ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33249 } } }
I20260812 00:14:13.915252 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:13.915481 7057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:13.916216 7057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:13.917730 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:13.913301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:58903 (local address 127.0.0.1:33249)
0812 00:14:13.913357 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:13.913359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:13.913376 (+ 17us) server_negotiation.cc:407] Connection header received
0812 00:14:13.913395 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:13.913395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:13.913403 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:13.913454 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:13.913709 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.914520 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.915685 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:13.915876 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:13.915971 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:13.916935 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:13.916937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:13.916937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:13.916943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:13.916959 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:13.916964 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:13.917013 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:13.917664 (+ 651us) server_negotiation.cc:299] Negotiation successful
0812 00:14:13.917675 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 00:14:13.918303 7057 catalog_manager.cc:1347] Generated new cluster ID: d7d7f6033739453ea7ac5361e6c61fd6
I20260812 00:14:13.918385 7057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:13.946592 7057 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:13.947459 7057 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:13.954821 7057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a45a19c99d9448cafbdf3855ad1a3ac: Generated new TSK 0
I20260812 00:14:13.955078 7057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:13.957882 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 6969
W20260812 00:14:13.965211 6953 connection.cc:535] client connection to 127.0.0.1:33249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33249): Connection reset by peer (error 104)
2026-08-12T00:14:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:14 dist-test-slave-zpfg krb5kdc[7074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:14 dist-test-slave-zpfg krb5kdc[7074](info): set up 2 sockets
Aug 12 00:14:14 dist-test-slave-zpfg krb5kdc[7074](info): commencing operation
krb5kdc: starting...
W20260812 00:14:15.067718 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:15 dist-test-slave-zpfg krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 00:14:15.209350 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:15.295267 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:15.295531 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:15.295571 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:15.296819 7087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:15.296895 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:15.296924 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:15.296947 7087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:15.297009 7087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:15.298702 7087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:15.298987 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:15.299234 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:15.305099 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:15.305094 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:15.305115 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:15.305505 7087 server_base.cc:1034] running on GCE node
I20260812 00:14:15.305817 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:15.306144 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:15.307322 7087 hybrid_clock.cc:648] HybridClock initialized: now 1786493655307295 us; error 47 us; skew 500 ppm
Aug 12 00:14:15 dist-test-slave-zpfg krb5kdc[7074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:15.310055 7087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 00:14:15.310581 7087 webserver.cc:460] Webserver started at http://127.0.0.1:40161/ using document root <none> and password file <none>
I20260812 00:14:15.310822 7087 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:15.310891 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:15.311029 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:15.312127 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "9f4150a35360412eba2444e94015b80e"
format_stamp: "Formatted at 2026-08-12 00:14:15 on dist-test-slave-zpfg"
server_key: "1dec01f7569f1459bbd722d3e13eb68d"
server_key_iv: "d408ca8c0ae615ea62c250f08eb3f286"
server_key_version: "encryptionkey@0"
I20260812 00:14:15.312525 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "9f4150a35360412eba2444e94015b80e"
format_stamp: "Formatted at 2026-08-12 00:14:15 on dist-test-slave-zpfg"
server_key: "1dec01f7569f1459bbd722d3e13eb68d"
server_key_iv: "d408ca8c0ae615ea62c250f08eb3f286"
server_key_version: "encryptionkey@0"
I20260812 00:14:15.314153 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:15.315099 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:15.315403 7087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:15.315541 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "9f4150a35360412eba2444e94015b80e"
format_stamp: "Formatted at 2026-08-12 00:14:15 on dist-test-slave-zpfg"
server_key: "1dec01f7569f1459bbd722d3e13eb68d"
server_key_iv: "d408ca8c0ae615ea62c250f08eb3f286"
server_key_version: "encryptionkey@0"
I20260812 00:14:15.315629 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:15.319746 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:15.322227 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:15.322453 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:15.326865 7087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43163
I20260812 00:14:15.326897 7157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43163 every 8 connection(s)
I20260812 00:14:15.327255 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:15.328171 7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:15.330588 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e: Bootstrap starting.
I20260812 00:14:15.331161 7158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:15.331398 7158 log.cc:824] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:15.332182 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e: No bootstrap required, opened a new log
I20260812 00:14:15.333822 7158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4150a35360412eba2444e94015b80e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43163 } }
I20260812 00:14:15.333964 7158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:15.333987 7158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4150a35360412eba2444e94015b80e, State: Initialized, Role: FOLLOWER
I20260812 00:14:15.334093 7158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4150a35360412eba2444e94015b80e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43163 } }
I20260812 00:14:15.334276 7158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:15.334326 7158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:15.334406 7158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:15.334955 7158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4150a35360412eba2444e94015b80e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43163 } }
I20260812 00:14:15.335083 7158 leader_election.cc:302] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f4150a35360412eba2444e94015b80e; no voters:
I20260812 00:14:15.335268 7158 leader_election.cc:288] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:15.335315 7162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:15.335541 7162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [term 1 LEADER]: Becoming Leader. State: Replica: 9f4150a35360412eba2444e94015b80e, State: Running, Role: LEADER
I20260812 00:14:15.335589 7158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:15.335705 7162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4150a35360412eba2444e94015b80e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43163 } }
I20260812 00:14:15.335779 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7087
I20260812 00:14:15.335919 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:15.336143 16783 external_mini_cluster.cc:1428] Setting key 37c62bdd7cb53e7391fd08f9cb149ca7
I20260812 00:14:15.336400 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f4150a35360412eba2444e94015b80e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4150a35360412eba2444e94015b80e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43163 } } }
I20260812 00:14:15.336484 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:15.336993 7164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f4150a35360412eba2444e94015b80e. Latest consensus state: current_term: 1 leader_uuid: "9f4150a35360412eba2444e94015b80e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f4150a35360412eba2444e94015b80e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43163 } } }
I20260812 00:14:15.337105 7164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e [sys.catalog]: This master's current role is: LEADER
W20260812 00:14:15.337757 7182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:15.337868 7182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:15.338210 7171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:15.338420 7171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:15.339167 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:15.336902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:59837 (local address 127.0.0.1:43163)
0812 00:14:15.337467 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:15.337477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:15.337525 (+ 48us) server_negotiation.cc:407] Connection header received
0812 00:14:15.337654 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:15.337662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:15.337694 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:15.337864 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:15.338681 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:15.338684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:15.338684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:15.338689 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:15.338741 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:15.338758 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:15.338839 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:15.338910 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 00:14:15.338954 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":65,"threads_started":1}
I20260812 00:14:15.340127 7171 catalog_manager.cc:1347] Generated new cluster ID: 669afcb981bb4e23a4e2362b0b2f98be
I20260812 00:14:15.340181 7171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:15.340163 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:15.342854 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:15.341616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.67:56457 (local address 127.0.0.1:43163)
0812 00:14:15.341655 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:15.341657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:15.341791 (+ 134us) server_negotiation.cc:407] Connection header received
0812 00:14:15.341869 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:15.341870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:15.341876 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:15.341927 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:15.342616 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:15.342618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:15.342619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:15.342624 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:15.342639 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:15.342643 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:15.342694 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:15.342755 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 00:14:15.342768 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:14:15.353681 7171 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:15.353942 7171 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:15.358737 7171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f4150a35360412eba2444e94015b80e: Generated new TSK 0
I20260812 00:14:15.358951 7171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:15.368026 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7087
2026-08-12T00:14:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:15 dist-test-slave-zpfg krb5kdc[7193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:15 dist-test-slave-zpfg krb5kdc[7193](info): set up 2 sockets
Aug 12 00:14:15 dist-test-slave-zpfg krb5kdc[7193](info): commencing operation
krb5kdc: starting...
W20260812 00:14:16.534557 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:16 dist-test-slave-zpfg krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:16.673751 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:44807
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:16.759874 7206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:16.760157 7206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:16.760202 7206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:16.761415 7206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:16.761497 7206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:16.761519 7206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:16.761538 7206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:16.761569 7206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:16.763180 7206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:44807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:16.763487 7206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:16.763738 7206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:16.769104 7212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:16.769117 7214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:16.769126 7211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:16.769495 7206 server_base.cc:1034] running on GCE node
I20260812 00:14:16.769766 7206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:16.770124 7206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:16.771337 7206 hybrid_clock.cc:648] HybridClock initialized: now 1786493656771311 us; error 60 us; skew 500 ppm
Aug 12 00:14:16 dist-test-slave-zpfg krb5kdc[7193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493656, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:16.779844 7206 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:16.780561 7206 webserver.cc:460] Webserver started at http://127.16.99.254:45359/ using document root <none> and password file <none>
I20260812 00:14:16.780846 7206 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:16.780920 7206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:16.781093 7206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:16.782766 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "bb9a9367277642cba48428c4ce57670f"
format_stamp: "Formatted at 2026-08-12 00:14:16 on dist-test-slave-zpfg"
server_key: "89cdb0161851aba3c5116b0f7065a370"
server_key_iv: "20f8a93a1482f80317aba5e375729c0e"
server_key_version: "encryptionkey@0"
I20260812 00:14:16.783171 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "bb9a9367277642cba48428c4ce57670f"
format_stamp: "Formatted at 2026-08-12 00:14:16 on dist-test-slave-zpfg"
server_key: "89cdb0161851aba3c5116b0f7065a370"
server_key_iv: "20f8a93a1482f80317aba5e375729c0e"
server_key_version: "encryptionkey@0"
I20260812 00:14:16.784655 7206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:16.785531 7221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:16.785744 7206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:16.785856 7206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "bb9a9367277642cba48428c4ce57670f"
format_stamp: "Formatted at 2026-08-12 00:14:16 on dist-test-slave-zpfg"
server_key: "89cdb0161851aba3c5116b0f7065a370"
server_key_iv: "20f8a93a1482f80317aba5e375729c0e"
server_key_version: "encryptionkey@0"
I20260812 00:14:16.785949 7206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:16.802717 7206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:16.804749 7206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:16.804958 7206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:16.809382 7206 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:44807
I20260812 00:14:16.809455 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:44807 every 8 connection(s)
I20260812 00:14:16.809854 7206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:16.810168 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7206
I20260812 00:14:16.810307 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:16.810592 16783 external_mini_cluster.cc:1428] Setting key a3e79a3c327b8189ef3b41255a4f895a
I20260812 00:14:16.811053 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:16.814754 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f: Bootstrap starting.
I20260812 00:14:16.815500 7275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:16.815806 7275 log.cc:824] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:16.816684 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f: No bootstrap required, opened a new log
Aug 12 00:14:16 dist-test-slave-zpfg krb5kdc[7193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:14:16.818910 7275 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9a9367277642cba48428c4ce57670f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44807 } }
I20260812 00:14:16.819108 7275 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:16.819164 7275 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9a9367277642cba48428c4ce57670f, State: Initialized, Role: FOLLOWER
I20260812 00:14:16.819329 7275 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9a9367277642cba48428c4ce57670f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44807 } }
I20260812 00:14:16.819425 7275 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:16.819458 7275 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:16.819517 7275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:16.820230 7275 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9a9367277642cba48428c4ce57670f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44807 } }
I20260812 00:14:16.820325 7275 leader_election.cc:302] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb9a9367277642cba48428c4ce57670f; no voters:
I20260812 00:14:16.820539 7275 leader_election.cc:288] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:16.820600 7280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:16.820801 7280 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [term 1 LEADER]: Becoming Leader. State: Replica: bb9a9367277642cba48428c4ce57670f, State: Running, Role: LEADER
I20260812 00:14:16.820838 7275 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:16.820977 7280 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9a9367277642cba48428c4ce57670f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44807 } }
I20260812 00:14:16.821367 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9a9367277642cba48428c4ce57670f. Latest consensus state: current_term: 1 leader_uuid: "bb9a9367277642cba48428c4ce57670f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9a9367277642cba48428c4ce57670f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44807 } } }
I20260812 00:14:16.821446 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:16.822005 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb9a9367277642cba48428c4ce57670f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb9a9367277642cba48428c4ce57670f" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 44807 } } }
I20260812 00:14:16.822198 7288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:16.822896 7288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:16.823298 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:16.811282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33559 (local address 127.16.99.254:44807)
0812 00:14:16.811526 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:16.811534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:16.811561 (+ 27us) server_negotiation.cc:407] Connection header received
0812 00:14:16.812283 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:16.812292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:16.812345 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:16.812465 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:16.813400 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:16.814541 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:16.815634 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:16.816162 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:16.816266 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:16.819030 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:16.819034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:16.819035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:16.819046 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:16.821196 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:16.821713 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:16.821716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:16.821716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:16.821825 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:16.822112 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:16.822112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:16.822113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:16.822483 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:16.822660 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:16.822959 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 00:14:16.823066 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260812 00:14:16.823796 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:16.824374 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:16.824887 7288 catalog_manager.cc:1347] Generated new cluster ID: 91e2cf76d6c4426a86e8a5b7442a15dc
I20260812 00:14:16.824944 7288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:16.832820 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:16.825912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57145 (local address 127.16.99.254:44807)
0812 00:14:16.825950 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:16.825951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:16.826512 (+ 561us) server_negotiation.cc:407] Connection header received
0812 00:14:16.826587 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:16.826588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:16.826598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:16.826654 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:16.827256 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:16.827753 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:16.828492 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:16.828726 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:16.828804 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:16.830066 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:16.830068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:16.830069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:16.830076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:16.831583 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:16.831999 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:16.832000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:16.832000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:16.832061 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:16.832294 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:16.832295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:16.832296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:16.832410 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:16.832523 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:16.832652 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 00:14:16.832716 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:14:16.840914 7288 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:16.841660 7288 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:16.846634 7288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb9a9367277642cba48428c4ce57670f: Generated new TSK 0
I20260812 00:14:16.846879 7288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:16.855682 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7206
W20260812 00:14:16.862226 7191 connection.cc:535] client connection to 127.16.99.254:44807 recv error: Network error: failed to read from TLS socket (remote: 127.16.99.254:44807): Connection reset by peer (error 104)
2026-08-12T00:14:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:16 dist-test-slave-zpfg krb5kdc[7307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:16 dist-test-slave-zpfg krb5kdc[7307](info): set up 2 sockets
Aug 12 00:14:16 dist-test-slave-zpfg krb5kdc[7307](info): commencing operation
krb5kdc: starting...
W20260812 00:14:17.951447 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:18 dist-test-slave-zpfg krb5kdc[7307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:18.093742 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38643
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:18.180200 7320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:18.180481 7320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:18.180521 7320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:18.181787 7320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:18.181865 7320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:18.181891 7320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:18.181926 7320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:18.181974 7320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:18.183638 7320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:38643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:18.183948 7320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:18.184207 7320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:18.189441 7325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:18.189502 7326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:18.189610 7320 server_base.cc:1034] running on GCE node
W20260812 00:14:18.189450 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:18.190081 7320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:18.190449 7320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:18.191638 7320 hybrid_clock.cc:648] HybridClock initialized: now 1786493658191613 us; error 42 us; skew 500 ppm
Aug 12 00:14:18 dist-test-slave-zpfg krb5kdc[7307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493658, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:18.194562 7320 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:18.195153 7320 webserver.cc:460] Webserver started at http://127.16.99.254:37509/ using document root <none> and password file <none>
I20260812 00:14:18.195473 7320 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:18.195564 7320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:18.195734 7320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:18.197176 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "b6c311c632d048b6afdf2ed60f0d0292"
format_stamp: "Formatted at 2026-08-12 00:14:18 on dist-test-slave-zpfg"
server_key: "9c9e948ce0dbc8c12c56d5294b72ff76"
server_key_iv: "abdc8b49dd8a31ae4fb79db7a7426949"
server_key_version: "encryptionkey@0"
I20260812 00:14:18.197650 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "b6c311c632d048b6afdf2ed60f0d0292"
format_stamp: "Formatted at 2026-08-12 00:14:18 on dist-test-slave-zpfg"
server_key: "9c9e948ce0dbc8c12c56d5294b72ff76"
server_key_iv: "abdc8b49dd8a31ae4fb79db7a7426949"
server_key_version: "encryptionkey@0"
I20260812 00:14:18.199288 7320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260812 00:14:18.200197 7336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:18.200546 7320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:18.200714 7320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "b6c311c632d048b6afdf2ed60f0d0292"
format_stamp: "Formatted at 2026-08-12 00:14:18 on dist-test-slave-zpfg"
server_key: "9c9e948ce0dbc8c12c56d5294b72ff76"
server_key_iv: "abdc8b49dd8a31ae4fb79db7a7426949"
server_key_version: "encryptionkey@0"
I20260812 00:14:18.200815 7320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:18.223358 7320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:18.225795 7320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:18.225993 7320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:18.230417 7320 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:38643
I20260812 00:14:18.230471 7390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:38643 every 8 connection(s)
I20260812 00:14:18.230890 7320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:18.231894 7391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:18.234289 7391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292: Bootstrap starting.
I20260812 00:14:18.234975 7391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:18.235284 7391 log.cc:824] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:18.236100 7391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292: No bootstrap required, opened a new log
I20260812 00:14:18.237716 7391 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c311c632d048b6afdf2ed60f0d0292" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38643 } }
I20260812 00:14:18.237927 7391 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:18.237951 7391 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c311c632d048b6afdf2ed60f0d0292, State: Initialized, Role: FOLLOWER
I20260812 00:14:18.238123 7391 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c311c632d048b6afdf2ed60f0d0292" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38643 } }
I20260812 00:14:18.238189 7391 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:18.238211 7391 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:18.238277 7391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:18.239296 7391 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c311c632d048b6afdf2ed60f0d0292" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38643 } }
I20260812 00:14:18.239395 7391 leader_election.cc:302] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c311c632d048b6afdf2ed60f0d0292; no voters:
I20260812 00:14:18.239614 7391 leader_election.cc:288] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:18.239814 7396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:18.239933 7391 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:18.239946 7396 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [term 1 LEADER]: Becoming Leader. State: Replica: b6c311c632d048b6afdf2ed60f0d0292, State: Running, Role: LEADER
I20260812 00:14:18.240053 7396 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c311c632d048b6afdf2ed60f0d0292" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38643 } }
I20260812 00:14:18.240475 7395 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c311c632d048b6afdf2ed60f0d0292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c311c632d048b6afdf2ed60f0d0292" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38643 } } }
I20260812 00:14:18.240754 7395 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:18.240780 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7320
I20260812 00:14:18.240914 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:18.240669 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c311c632d048b6afdf2ed60f0d0292. Latest consensus state: current_term: 1 leader_uuid: "b6c311c632d048b6afdf2ed60f0d0292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c311c632d048b6afdf2ed60f0d0292" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38643 } } }
I20260812 00:14:18.241009 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:18.241127 16783 external_mini_cluster.cc:1428] Setting key b6b4bea6caf1e2eb067cff036158d55c
W20260812 00:14:18.242048 7412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:18.242127 7412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:18.242220 7405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:18.242465 7405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:18.244159 7405 catalog_manager.cc:1347] Generated new cluster ID: 5a7ae43fbae14f4f91d3fcecd2e13a94
I20260812 00:14:18.244222 7405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:18.247677 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:18.241666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46883 (local address 127.16.99.254:38643)
0812 00:14:18.241870 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:18.241881 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:18.241918 (+ 37us) server_negotiation.cc:407] Connection header received
0812 00:14:18.242049 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:18.242056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:18.242114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:18.242271 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:18.242657 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:18.244026 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:18.245193 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:18.245889 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:18.246022 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:18.246892 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:18.246896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:18.246896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:18.247138 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:18.247181 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:18.247197 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:18.247297 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:18.247397 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 00:14:18.247425 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260812 00:14:18.248795 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:18.254513 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:18.250625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50809 (local address 127.16.99.254:38643)
0812 00:14:18.250670 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:18.250672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:18.250734 (+ 62us) server_negotiation.cc:407] Connection header received
0812 00:14:18.250805 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:18.250806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:18.250816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:18.250873 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:18.251029 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:18.251893 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:18.252927 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:18.253286 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:18.253356 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:18.254181 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:18.254182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:18.254183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:18.254188 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:18.254204 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:18.254210 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:18.254274 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:18.254408 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 00:14:18.254422 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:18.256546 7405 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:18.257306 7405 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:18.269963 7405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c311c632d048b6afdf2ed60f0d0292: Generated new TSK 0
I20260812 00:14:18.270229 7405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:18.277041 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7320
W20260812 00:14:18.284662 7305 connection.cc:535] client connection to 127.16.99.254:38643 recv error: Network error: failed to read from TLS socket (remote: 127.16.99.254:38643): Connection reset by peer (error 104)
2026-08-12T00:14:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:18 dist-test-slave-zpfg krb5kdc[7425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:18 dist-test-slave-zpfg krb5kdc[7425](info): set up 2 sockets
Aug 12 00:14:18 dist-test-slave-zpfg krb5kdc[7425](info): commencing operation
krb5kdc: starting...
W20260812 00:14:19.366852 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:19 dist-test-slave-zpfg krb5kdc[7425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:19.505199 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37041
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:37041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:19.601485 7439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:19.601788 7439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:19.601838 7439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:19.603271 7439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:19.603356 7439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:19.603384 7439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:19.603417 7439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:19.603453 7439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:19.605201 7439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:37041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:37041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:19.605535 7439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:19.605815 7439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:19.611192 7448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:19.611241 7445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:19.611253 7446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:19.611516 7439 server_base.cc:1034] running on GCE node
I20260812 00:14:19.611850 7439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:19.612221 7439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:19.613391 7439 hybrid_clock.cc:648] HybridClock initialized: now 1786493659613354 us; error 52 us; skew 500 ppm
Aug 12 00:14:19 dist-test-slave-zpfg krb5kdc[7425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493659, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:19.616410 7439 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:19.616989 7439 webserver.cc:460] Webserver started at http://127.16.99.254:44827/ using document root <none> and password file <none>
I20260812 00:14:19.617280 7439 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:19.617362 7439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:19.617533 7439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:19.618868 7439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "c0a0b49a88894b02948815f700709068"
format_stamp: "Formatted at 2026-08-12 00:14:19 on dist-test-slave-zpfg"
server_key: "01a748ea908e63b00e6825cc378968d5"
server_key_iv: "75368fe31d8c96eccd7373edf44114a1"
server_key_version: "encryptionkey@0"
I20260812 00:14:19.619288 7439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "c0a0b49a88894b02948815f700709068"
format_stamp: "Formatted at 2026-08-12 00:14:19 on dist-test-slave-zpfg"
server_key: "01a748ea908e63b00e6825cc378968d5"
server_key_iv: "75368fe31d8c96eccd7373edf44114a1"
server_key_version: "encryptionkey@0"
I20260812 00:14:19.620815 7439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:19.621769 7455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:19.621966 7439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:19.622082 7439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "c0a0b49a88894b02948815f700709068"
format_stamp: "Formatted at 2026-08-12 00:14:19 on dist-test-slave-zpfg"
server_key: "01a748ea908e63b00e6825cc378968d5"
server_key_iv: "75368fe31d8c96eccd7373edf44114a1"
server_key_version: "encryptionkey@0"
I20260812 00:14:19.622176 7439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:19.626029 7439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:19.628105 7439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:19.628310 7439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:19.633605 7439 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:37041
I20260812 00:14:19.633657 7511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:37041 every 8 connection(s)
I20260812 00:14:19.634025 7439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:19.635124 7512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:19.637450 7512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068: Bootstrap starting.
I20260812 00:14:19.638043 7512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:19.638375 7512 log.cc:824] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:19.639350 7512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068: No bootstrap required, opened a new log
I20260812 00:14:19.641176 7512 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0b49a88894b02948815f700709068" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37041 } }
I20260812 00:14:19.641340 7512 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:19.641367 7512 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a0b49a88894b02948815f700709068, State: Initialized, Role: FOLLOWER
I20260812 00:14:19.641549 7512 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0b49a88894b02948815f700709068" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37041 } }
I20260812 00:14:19.641616 7512 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:19.641682 7512 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:19.641744 7512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:19.642132 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7439
I20260812 00:14:19.642272 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:19.642500 16783 external_mini_cluster.cc:1428] Setting key 2b8d62c0baa4499a24420fe61da342ff
I20260812 00:14:19.642992 7512 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0b49a88894b02948815f700709068" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37041 } }
I20260812 00:14:19.643191 7512 leader_election.cc:302] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0a0b49a88894b02948815f700709068; no voters:
I20260812 00:14:19.643471 7512 leader_election.cc:288] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:19.643493 7517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:19.643687 7517 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [term 1 LEADER]: Becoming Leader. State: Replica: c0a0b49a88894b02948815f700709068, State: Running, Role: LEADER
I20260812 00:14:19.643828 7512 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:19.643874 7517 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0b49a88894b02948815f700709068" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37041 } }
I20260812 00:14:19.644575 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0a0b49a88894b02948815f700709068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0b49a88894b02948815f700709068" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37041 } } }
I20260812 00:14:19.644776 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:19.645074 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0a0b49a88894b02948815f700709068. Latest consensus state: current_term: 1 leader_uuid: "c0a0b49a88894b02948815f700709068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0b49a88894b02948815f700709068" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 37041 } } }
I20260812 00:14:19.645171 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:19.645260 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:19.643070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33363 (local address 127.16.99.254:37041)
0812 00:14:19.643265 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:19.643275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:19.643318 (+ 43us) server_negotiation.cc:407] Connection header received
0812 00:14:19.643446 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:19.643453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:19.643477 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:19.643686 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:19.644425 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:19.644430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:19.644430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:19.644435 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:19.644488 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:19.644500 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:19.644936 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:19.644976 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 00:14:19.645023 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260812 00:14:19.646152 7526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:19.646287 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:19.646827 7526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:19.648522 7526 catalog_manager.cc:1347] Generated new cluster ID: 4b2eda96905846a0aaacbda578773b9f
I20260812 00:14:19.648592 7526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:19.652159 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:19.650797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39289 (local address 127.16.99.254:37041)
0812 00:14:19.650840 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:19.650842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:19.651015 (+ 173us) server_negotiation.cc:407] Connection header received
0812 00:14:19.651077 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:19.651078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:19.651098 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:19.651161 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:19.651907 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:19.651908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:19.651909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:19.651913 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:19.651930 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:19.651934 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:19.652002 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:19.652075 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 00:14:19.652089 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:19.658648 7526 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:19.658895 7526 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:19.662770 7526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0a0b49a88894b02948815f700709068: Generated new TSK 0
I20260812 00:14:19.662986 7526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:19.664749 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7439
2026-08-12T00:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:19 dist-test-slave-zpfg krb5kdc[7545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:19 dist-test-slave-zpfg krb5kdc[7545](info): set up 2 sockets
Aug 12 00:14:19 dist-test-slave-zpfg krb5kdc[7545](info): commencing operation
krb5kdc: starting...
W20260812 00:14:20.842765 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:20 dist-test-slave-zpfg krb5kdc[7545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:20.982766 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40233
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:21.072338 7558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:21.072598 7558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:21.072641 7558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:21.073872 7558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:21.073944 7558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:21.073971 7558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:21.073992 7558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:21.074021 7558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:21.075664 7558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:21.075964 7558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:21.076220 7558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:21.081617 7564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:21.081702 7558 server_base.cc:1034] running on GCE node
W20260812 00:14:21.081666 7565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:21.081682 7567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:21.082206 7558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:21.082609 7558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:21.083813 7558 hybrid_clock.cc:648] HybridClock initialized: now 1786493661083787 us; error 38 us; skew 500 ppm
Aug 12 00:14:21 dist-test-slave-zpfg krb5kdc[7545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493661, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:21.086840 7558 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:21.087361 7558 webserver.cc:460] Webserver started at http://127.16.99.254:36213/ using document root <none> and password file <none>
I20260812 00:14:21.087633 7558 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:21.087715 7558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:21.087883 7558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:21.089164 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "bf1493b846054f9cbfb4beba33d43128"
format_stamp: "Formatted at 2026-08-12 00:14:21 on dist-test-slave-zpfg"
server_key: "79bc8c1ba1b46415a4bfbd5cc41f7be8"
server_key_iv: "dc9cc2695421eaea49ce7fc0ddfbd3eb"
server_key_version: "encryptionkey@0"
I20260812 00:14:21.089581 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "bf1493b846054f9cbfb4beba33d43128"
format_stamp: "Formatted at 2026-08-12 00:14:21 on dist-test-slave-zpfg"
server_key: "79bc8c1ba1b46415a4bfbd5cc41f7be8"
server_key_iv: "dc9cc2695421eaea49ce7fc0ddfbd3eb"
server_key_version: "encryptionkey@0"
I20260812 00:14:21.091259 7558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260812 00:14:21.092133 7575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:21.092326 7558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:21.092455 7558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "bf1493b846054f9cbfb4beba33d43128"
format_stamp: "Formatted at 2026-08-12 00:14:21 on dist-test-slave-zpfg"
server_key: "79bc8c1ba1b46415a4bfbd5cc41f7be8"
server_key_iv: "dc9cc2695421eaea49ce7fc0ddfbd3eb"
server_key_version: "encryptionkey@0"
I20260812 00:14:21.092561 7558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:21.121235 7558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:21.123622 7558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:21.123828 7558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:21.128749 7558 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:40233
I20260812 00:14:21.128788 7630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:40233 every 8 connection(s)
I20260812 00:14:21.129205 7558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:21.129695 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7558
I20260812 00:14:21.129843 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:21.130107 16783 external_mini_cluster.cc:1428] Setting key 5396a6318b9e4e3f8e959776ee3551c2
I20260812 00:14:21.130682 7631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:21.134980 7631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128: Bootstrap starting.
I20260812 00:14:21.135731 7631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:21.136145 7631 log.cc:824] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:21.137143 7631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128: No bootstrap required, opened a new log
Aug 12 00:14:21 dist-test-slave-zpfg krb5kdc[7545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:14:21.140034 7631 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1493b846054f9cbfb4beba33d43128" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40233 } }
I20260812 00:14:21.140281 7631 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:21.140370 7631 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1493b846054f9cbfb4beba33d43128, State: Initialized, Role: FOLLOWER
I20260812 00:14:21.140561 7631 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1493b846054f9cbfb4beba33d43128" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40233 } }
I20260812 00:14:21.140653 7631 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:21.140681 7631 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:21.140749 7631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:21.141435 7631 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1493b846054f9cbfb4beba33d43128" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40233 } }
I20260812 00:14:21.141534 7631 leader_election.cc:302] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf1493b846054f9cbfb4beba33d43128; no voters:
I20260812 00:14:21.141798 7631 leader_election.cc:288] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:21.141835 7636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:21.141960 7636 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [term 1 LEADER]: Becoming Leader. State: Replica: bf1493b846054f9cbfb4beba33d43128, State: Running, Role: LEADER
I20260812 00:14:21.142132 7631 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:21.142117 7636 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1493b846054f9cbfb4beba33d43128" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40233 } }
I20260812 00:14:21.142808 7637 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf1493b846054f9cbfb4beba33d43128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1493b846054f9cbfb4beba33d43128" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40233 } } }
I20260812 00:14:21.142823 7638 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf1493b846054f9cbfb4beba33d43128. Latest consensus state: current_term: 1 leader_uuid: "bf1493b846054f9cbfb4beba33d43128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1493b846054f9cbfb4beba33d43128" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 40233 } } }
I20260812 00:14:21.143030 7638 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:21.143013 7637 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:21.144007 7642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:21.144733 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:21.130877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60689 (local address 127.16.99.254:40233)
0812 00:14:21.131101 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:21.131115 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:21.131150 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:14:21.131627 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:21.131634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:21.131686 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:21.131822 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:21.132881 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:21.133915 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:21.135234 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:21.135742 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:21.135904 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:21.139250 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:21.139257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:21.139258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:21.139279 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:21.141818 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:21.142514 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:21.142517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:21.142518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:21.142641 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:21.142898 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:21.142899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:21.142900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:21.143412 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:21.143781 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:21.143970 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 00:14:21.144369 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260812 00:14:21.144734 7642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:21.145967 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:21.146689 7642 catalog_manager.cc:1347] Generated new cluster ID: 9787f71ec717478eaf5f8855768c1290
I20260812 00:14:21.146764 7642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:21.156291 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:21.147689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34653 (local address 127.16.99.254:40233)
0812 00:14:21.147739 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:21.147741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:21.147871 (+ 130us) server_negotiation.cc:407] Connection header received
0812 00:14:21.147909 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:21.147910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:21.147922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:21.148010 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:21.148708 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:21.149553 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:21.150685 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:21.151050 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:21.151136 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:21.152819 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:21.152821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:21.152822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:21.152830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:21.154853 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:21.155327 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:21.155329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:21.155329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:21.155401 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:21.155648 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:21.155649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:21.155650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:21.155821 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:21.155938 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:21.156088 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 00:14:21.156180 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 00:14:21.177836 7642 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:21.178668 7642 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:21.183296 7642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf1493b846054f9cbfb4beba33d43128: Generated new TSK 0
I20260812 00:14:21.183544 7642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:21.195346 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7558
2026-08-12T00:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1531 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:21 dist-test-slave-zpfg krb5kdc[7662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:21 dist-test-slave-zpfg krb5kdc[7662](info): set up 2 sockets
Aug 12 00:14:21 dist-test-slave-zpfg krb5kdc[7662](info): commencing operation
krb5kdc: starting...
W20260812 00:14:22.319342 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:22 dist-test-slave-zpfg krb5kdc[7662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:22.469980 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:41865
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:41865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:22.560631 7675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:22.560882 7675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:22.560909 7675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:22.562059 7675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:22.562105 7675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:22.562121 7675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:22.562134 7675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:22.562146 7675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:22.563656 7675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:41865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:41865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:22.563874 7675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:22.564056 7675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:22.569200 7681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.569206 7680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:22.569191 7683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:22.569371 7675 server_base.cc:1034] running on GCE node
I20260812 00:14:22.569626 7675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:22.569890 7675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:22.571036 7675 hybrid_clock.cc:648] HybridClock initialized: now 1786493662571014 us; error 42 us; skew 500 ppm
Aug 12 00:14:22 dist-test-slave-zpfg krb5kdc[7662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493662, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:22.573711 7675 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:22.574139 7675 webserver.cc:460] Webserver started at http://127.16.99.254:38835/ using document root <none> and password file <none>
I20260812 00:14:22.574381 7675 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:22.574445 7675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:22.574541 7675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:22.575591 7675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "f6fbd3f417014776b29a74d31ef2b8b5"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zpfg"
server_key: "8d0c166edcaaf1d2e5af436507170d38"
server_key_iv: "60f06a9c7a0eeb27ba5a06e20107700b"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.575912 7675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "f6fbd3f417014776b29a74d31ef2b8b5"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zpfg"
server_key: "8d0c166edcaaf1d2e5af436507170d38"
server_key_iv: "60f06a9c7a0eeb27ba5a06e20107700b"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.577301 7675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:22.578270 7690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:22.578533 7675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:22.578655 7675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "f6fbd3f417014776b29a74d31ef2b8b5"
format_stamp: "Formatted at 2026-08-12 00:14:22 on dist-test-slave-zpfg"
server_key: "8d0c166edcaaf1d2e5af436507170d38"
server_key_iv: "60f06a9c7a0eeb27ba5a06e20107700b"
server_key_version: "encryptionkey@0"
I20260812 00:14:22.578770 7675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:22.587478 7675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:22.590003 7675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:22.590188 7675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:22.594499 7675 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:41865
I20260812 00:14:22.594535 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:41865 every 8 connection(s)
I20260812 00:14:22.594921 7675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:22.595865 7745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:22.598155 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7675
I20260812 00:14:22.598278 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:22.598513 16783 external_mini_cluster.cc:1428] Setting key a7263c44f680dbf8cf85694f2d3d2712
I20260812 00:14:22.598701 7745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5: Bootstrap starting.
I20260812 00:14:22.599507 7745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:22.599821 7745 log.cc:824] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:22.600528 7745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5: No bootstrap required, opened a new log
I20260812 00:14:22.602447 7745 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41865 } }
I20260812 00:14:22.602589 7745 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:22.602612 7745 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fbd3f417014776b29a74d31ef2b8b5, State: Initialized, Role: FOLLOWER
I20260812 00:14:22.602780 7745 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41865 } }
I20260812 00:14:22.602856 7745 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:22.602878 7745 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:22.602948 7745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:22.603811 7745 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41865 } }
I20260812 00:14:22.603915 7745 leader_election.cc:302] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6fbd3f417014776b29a74d31ef2b8b5; no voters:
I20260812 00:14:22.604159 7745 leader_election.cc:288] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:22.604184 7750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:22.604398 7750 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [term 1 LEADER]: Becoming Leader. State: Replica: f6fbd3f417014776b29a74d31ef2b8b5, State: Running, Role: LEADER
I20260812 00:14:22.604516 7745 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:22.604564 7750 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41865 } }
I20260812 00:14:22.604585 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.599114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54139 (local address 127.16.99.254:41865)
0812 00:14:22.599312 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.599324 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.599359 (+ 35us) server_negotiation.cc:407] Connection header received
0812 00:14:22.599965 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.599973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.600025 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.600179 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:22.600426 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.601396 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.602246 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.602830 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.602964 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.603788 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:22.603792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:22.603792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:22.604049 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:22.604096 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:22.604107 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:22.604182 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.604262 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.604344 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260812 00:14:22.604933 7750 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41865 } } }
I20260812 00:14:22.605099 7750 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:22.605286 7750 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6fbd3f417014776b29a74d31ef2b8b5. Latest consensus state: current_term: 1 leader_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6fbd3f417014776b29a74d31ef2b8b5" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 41865 } } }
I20260812 00:14:22.605408 7750 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:22.605683 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 00:14:22.607044 7766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:22.607106 7766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:22.607223 7759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:22.607455 7759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:22.609196 7759 catalog_manager.cc:1347] Generated new cluster ID: 087cd43a9f274eaf9e01bf414d6b4c3b
I20260812 00:14:22.609253 7759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:22.611845 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:22.607306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47663 (local address 127.16.99.254:41865)
0812 00:14:22.607359 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:22.607361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:22.608351 (+ 990us) server_negotiation.cc:407] Connection header received
0812 00:14:22.608383 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:22.608385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:22.608396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:22.608465 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:22.608653 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.609379 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.610463 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:22.610769 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:22.610842 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:22.611599 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:22.611600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:22.611600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:22.611605 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:22.611623 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:22.611628 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:22.611689 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:22.611744 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 00:14:22.611767 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 00:14:22.617746 7759 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:22.618388 7759 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:22.626754 7759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6fbd3f417014776b29a74d31ef2b8b5: Generated new TSK 0
I20260812 00:14:22.626991 7759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:22.629911 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7675
2026-08-12T00:14:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:22 dist-test-slave-zpfg krb5kdc[7777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:22 dist-test-slave-zpfg krb5kdc[7777](info): set up 2 sockets
Aug 12 00:14:22 dist-test-slave-zpfg krb5kdc[7777](info): commencing operation
krb5kdc: starting...
W20260812 00:14:23.743429 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:23 dist-test-slave-zpfg krb5kdc[7777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:23.882017 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38685
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:23.970662 7790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:23.970928 7790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:23.970971 7790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:23.972178 7790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:23.972256 7790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:23.972281 7790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:23.972316 7790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:23.972360 7790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:23.973914 7790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:38685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:23.974231 7790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:23.974519 7790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:23.980125 7797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:23.980120 7796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.980481 7790 server_base.cc:1034] running on GCE node
W20260812 00:14:23.980163 7799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:23.980882 7790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:23.981204 7790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:23.982467 7790 hybrid_clock.cc:648] HybridClock initialized: now 1786493663982430 us; error 48 us; skew 500 ppm
Aug 12 00:14:23 dist-test-slave-zpfg krb5kdc[7777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493663, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:23.985513 7790 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:23.986207 7790 webserver.cc:460] Webserver started at http://127.16.99.254:45457/ using document root <none> and password file <none>
I20260812 00:14:23.986629 7790 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:23.986690 7790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:23.986832 7790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:23.988075 7790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e"
format_stamp: "Formatted at 2026-08-12 00:14:23 on dist-test-slave-zpfg"
server_key: "ec9467d94b3291a90f143ce50c795746"
server_key_iv: "29e78b758cf8cd4cc5e1c8c90073b46b"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.988464 7790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e"
format_stamp: "Formatted at 2026-08-12 00:14:23 on dist-test-slave-zpfg"
server_key: "ec9467d94b3291a90f143ce50c795746"
server_key_iv: "29e78b758cf8cd4cc5e1c8c90073b46b"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.989916 7790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 00:14:23.990933 7806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:23.991145 7790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 00:14:23.991355 7790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e"
format_stamp: "Formatted at 2026-08-12 00:14:23 on dist-test-slave-zpfg"
server_key: "ec9467d94b3291a90f143ce50c795746"
server_key_iv: "29e78b758cf8cd4cc5e1c8c90073b46b"
server_key_version: "encryptionkey@0"
I20260812 00:14:23.991441 7790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:23.995051 7790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:23.997603 7790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:23.997824 7790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:24.002410 7790 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:38685
I20260812 00:14:24.002501 7858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:38685 every 8 connection(s)
I20260812 00:14:24.002871 7790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:24.004114 7859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:24.006996 7859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e: Bootstrap starting.
I20260812 00:14:24.007659 7859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:24.007961 7859 log.cc:824] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:24.008558 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7790
I20260812 00:14:24.008689 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:24.008940 16783 external_mini_cluster.cc:1428] Setting key c6be4df36118bb83253e16cf26537d6c
I20260812 00:14:24.008989 7859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e: No bootstrap required, opened a new log
I20260812 00:14:24.011821 7859 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38685 } }
I20260812 00:14:24.012115 7859 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 00:14:24.012189 7859 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8baa53b4d2b4b8ab1139f5926d4de1e, State: Initialized, Role: FOLLOWER
I20260812 00:14:24.012399 7859 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38685 } }
I20260812 00:14:24.012532 7859 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 00:14:24.012584 7859 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 00:14:24.012607 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.009642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55031 (local address 127.16.99.254:38685)
0812 00:14:24.009903 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:24.009912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:24.009940 (+ 28us) server_negotiation.cc:407] Connection header received
0812 00:14:24.010657 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:24.010666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:24.010699 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:24.010879 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:24.011732 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:24.011737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:24.011737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:24.011741 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:24.011826 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:24.011838 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:24.011900 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:24.012012 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 00:14:24.012072 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260812 00:14:24.012642 7859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:24.013669 7859 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38685 } }
I20260812 00:14:24.013813 7859 leader_election.cc:302] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8baa53b4d2b4b8ab1139f5926d4de1e; no voters:
I20260812 00:14:24.014020 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:24.014043 7859 leader_election.cc:288] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 00:14:24.014118 7864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:24.014323 7859 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:24.014506 7864 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [term 1 LEADER]: Becoming Leader. State: Replica: d8baa53b4d2b4b8ab1139f5926d4de1e, State: Running, Role: LEADER
I20260812 00:14:24.014649 7864 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38685 } }
I20260812 00:14:24.015076 7865 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38685 } } }
I20260812 00:14:24.015168 7865 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:24.015352 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8baa53b4d2b4b8ab1139f5926d4de1e. Latest consensus state: current_term: 1 leader_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8baa53b4d2b4b8ab1139f5926d4de1e" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 38685 } } }
I20260812 00:14:24.015414 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:24.015823 7873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:24.016448 7873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:24.018530 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:24.016471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40853 (local address 127.16.99.254:38685)
0812 00:14:24.016513 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:24.016515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:24.016916 (+ 401us) server_negotiation.cc:407] Connection header received
0812 00:14:24.016996 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:24.016997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:24.017023 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:24.017107 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:24.018132 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:24.018134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:24.018134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 00:14:24.018141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:24.018162 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 00:14:24.018168 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:24.018224 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:24.018430 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 00:14:24.018446 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:14:24.018869 7873 catalog_manager.cc:1347] Generated new cluster ID: 8ac359e1564e4c3ea252b754cd8b2b1e
I20260812 00:14:24.018958 7873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:24.033973 7873 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:24.034209 7873 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:24.039440 7873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8baa53b4d2b4b8ab1139f5926d4de1e: Generated new TSK 0
I20260812 00:14:24.039698 7873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:24.048682 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7790
2026-08-12T00:14:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1416 ms)
[----------] 18 tests from AuthTokenIssuingTest (25994 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:24 dist-test-slave-zpfg krb5kdc[7891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:24 dist-test-slave-zpfg krb5kdc[7891](info): set up 2 sockets
Aug 12 00:14:24 dist-test-slave-zpfg krb5kdc[7891](info): commencing operation
krb5kdc: starting...
W20260812 00:14:25.192111 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:25.338075 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45273
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:45273,127.16.99.253:33897,127.16.99.252:44087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:25.427043 7904 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:14:25.427280 7904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:25.427299 7904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:25.427321 7904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:25.428503 7904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:25.428553 7904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:25.428568 7904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:25.428580 7904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:25.428592 7904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:25.430284 7904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:45273,127.16.99.253:33897,127.16.99.252:44087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:25.430631 7904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:25.430912 7904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:25.436436 7909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:25.436437 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:25.436481 7910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:25.436878 7904 server_base.cc:1034] running on GCE node
I20260812 00:14:25.437170 7904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:25.437451 7904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:25.438635 7904 hybrid_clock.cc:648] HybridClock initialized: now 1786493665438618 us; error 40 us; skew 500 ppm
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:25.441512 7904 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:25.442006 7904 webserver.cc:460] Webserver started at http://127.16.99.254:38743/ using document root <none> and password file <none>
I20260812 00:14:25.442258 7904 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:25.442329 7904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:25.442515 7904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:25.443629 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "9a8807723d454611b2d1e26bbffa9343"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "a862b1d95e5e48d96e9efa66eef8f4d2"
server_key_iv: "30c795417327d0264ab1474fff739c07"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.444023 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "9a8807723d454611b2d1e26bbffa9343"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "a862b1d95e5e48d96e9efa66eef8f4d2"
server_key_iv: "30c795417327d0264ab1474fff739c07"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.445456 7904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:25.446285 7920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:25.446539 7904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:25.446640 7904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "9a8807723d454611b2d1e26bbffa9343"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "a862b1d95e5e48d96e9efa66eef8f4d2"
server_key_iv: "30c795417327d0264ab1474fff739c07"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.446703 7904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:25.454200 7904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:25.456682 7904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:25.456815 7904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:25.460795 7904 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:45273
I20260812 00:14:25.460834 7977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:45273 every 8 connection(s)
I20260812 00:14:25.461203 7904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:25.462255 7978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:25.463922 7978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:25.465087 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7904
I20260812 00:14:25.465207 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:25.465468 16783 external_mini_cluster.cc:1428] Setting key 82489bf3747462f344b4d04cc4d2def8
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.253@KRBTEST.COM; defaulting to no policy
I20260812 00:14:25.476652 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.464670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45273 (local address 127.0.0.1:58834)
0812 00:14:25.464864 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.464903 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.465572 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.465918 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.465920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.466179 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.466479 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.466484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.467963 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.467965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.468985 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.468988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.469650 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.469652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.469715 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.470470 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.470479 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.472316 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.474858 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.474866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.474868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.475137 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.475486 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.475487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.475487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.475574 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.475983 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.475989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.476057 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.476180 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.476436 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":100,"thread_start_us":108,"threads_started":1}
I20260812 00:14:25.476652 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.464694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.16.99.254:45273)
0812 00:14:25.464890 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.464901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.465019 (+ 118us) server_negotiation.cc:407] Connection header received
0812 00:14:25.465759 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.465762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.465787 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.465901 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.466584 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.467856 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.469087 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.469507 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.469656 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.472436 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.472447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.472447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.472456 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.474748 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.475249 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.475252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.475253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.475402 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.475646 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.475647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.475647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.475896 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.476025 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.476220 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.476370 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":122,"threads_started":1}
I20260812 00:14:25.477566 7978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:25.477998 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.477842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51412)
0812 00:14:25.477896 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.477965 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 00:14:25.478113 7921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.253@KRBTEST.COM" created.
I20260812 00:14:25.478657 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.478557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51422)
0812 00:14:25.478604 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.478629 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 00:14:25.478761 7978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:33897: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab.
Entry for principal kudu/127.16.99.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab.
I20260812 00:14:25.497838 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:33897
--webserver_interface=127.16.99.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab
--principal=kudu/127.16.99.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:45273,127.16.99.253:33897,127.16.99.252:44087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:25.528016 7978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } attempt: 1
I20260812 00:14:25.528620 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.528456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51432)
0812 00:14:25.528526 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.528578 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 00:14:25.529142 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.529037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51448)
0812 00:14:25.529086 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.529118 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:14:25.529443 7978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:33897: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
W20260812 00:14:25.584364 7985 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:14:25.584596 7985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:25.584616 7985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:25.584638 7985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:25.585759 7985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:25.585803 7985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:25.585819 7985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:25.585831 7985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:25.585843 7985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:25.587340 7985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:45273,127.16.99.253:33897,127.16.99.252:44087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab
--principal=kudu/127.16.99.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.99.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:25.587558 7985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:25.587739 7985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:25.592808 7993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:25.592844 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:25.592875 7995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:25.593245 7985 server_base.cc:1034] running on GCE node
I20260812 00:14:25.593547 7985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:25.593856 7985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:25.595033 7985 hybrid_clock.cc:648] HybridClock initialized: now 1786493665595026 us; error 37 us; skew 500 ppm
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:25.597591 7978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } attempt: 2
I20260812 00:14:25.597848 7985 init.cc:377] Logged in from keytab as kudu/127.16.99.253@KRBTEST.COM (short username kudu)
I20260812 00:14:25.597991 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.597854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51464)
0812 00:14:25.597915 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.597948 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 00:14:25.598490 7985 webserver.cc:460] Webserver started at http://127.16.99.253:45933/ using document root <none> and password file <none>
I20260812 00:14:25.598492 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.598388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51470)
0812 00:14:25.598427 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.598464 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 00:14:25.598587 7978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:33897: Network error: Client connection negotiation failed: client connection to 127.16.99.253:33897: connect: Connection refused (error 111)
I20260812 00:14:25.598814 7985 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:25.598866 7985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:25.598963 7985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:25.600114 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data/instance:
uuid: "42991915533349f08da28c3c573a0aef"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "869ba69b8de059c0c84959cfa3afcc7b"
server_key_iv: "75ea8ee094d68476e4288abc077be4d7"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.600436 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/wal/instance:
uuid: "42991915533349f08da28c3c573a0aef"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "869ba69b8de059c0c84959cfa3afcc7b"
server_key_iv: "75ea8ee094d68476e4288abc077be4d7"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.601720 7985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:25.602519 8002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:25.602679 7985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:25.602774 7985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/wal
uuid: "42991915533349f08da28c3c573a0aef"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "869ba69b8de059c0c84959cfa3afcc7b"
server_key_iv: "75ea8ee094d68476e4288abc077be4d7"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.602833 7985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:25.637344 7985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:25.639923 7985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:25.640045 7985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:25.644246 7985 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.253:33897
I20260812 00:14:25.644356 8055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.253:33897 every 8 connection(s)
I20260812 00:14:25.644699 7985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
I20260812 00:14:25.645895 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:25.647547 8056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:14:25.654627 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 7985
I20260812 00:14:25.654729 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-1/wal/instance
I20260812 00:14:25.654956 16783 external_mini_cluster.cc:1428] Setting key acb18cb1a7ca73eae26373e58985e651
I20260812 00:14:25.657663 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.648373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.16.99.254:45273)
0812 00:14:25.648483 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.648486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.648747 (+ 261us) server_negotiation.cc:407] Connection header received
0812 00:14:25.649341 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.649343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.649358 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.649425 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.650141 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.650782 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.651784 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.652006 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.652097 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.654541 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.654542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.654543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.654553 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.656154 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.656616 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.656617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.656618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.656683 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.656958 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.656959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.656959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.657084 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.657159 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.657482 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.657570 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 00:14:25.657955 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.648319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45273 (local address 127.0.0.1:58838)
0812 00:14:25.648547 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.648595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.649168 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.649518 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.649521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.649835 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.650037 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.650041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.650877 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.650879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.651687 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.651689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.652129 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.652130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.652255 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.652896 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.652905 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.654435 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.656272 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.656282 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.656284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.656533 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.656757 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.656757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.656758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.656883 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.657151 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.657158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.657328 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.657446 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.657620 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260812 00:14:25.658763 8056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.99.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.252@KRBTEST.COM" created.
I20260812 00:14:25.667109 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.659082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51478)
0812 00:14:25.659133 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.659144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.659182 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.659505 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.659506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.659697 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.659779 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.659781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.660637 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.660638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.661529 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.661532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.662038 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.662040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.662127 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.662671 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.662676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.663977 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.665902 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.665903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.665903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.666074 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.666387 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.666388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.666388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.666450 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.666770 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.666771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.666819 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.666884 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.666984 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 00:14:25.667135 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.659184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.16.99.253:33897)
0812 00:14:25.659340 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.659343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.659359 (+ 16us) server_negotiation.cc:407] Connection header received
0812 00:14:25.659375 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.659376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.659386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.659478 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.659861 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.660536 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.661622 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.661958 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.662013 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.664069 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.664080 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.664080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.664087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.665798 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.666171 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.666174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.666174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.666233 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.666539 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.666540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.666541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.666700 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.666794 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.666949 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.667037 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260812 00:14:25.667963 8056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:25.668326 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.668196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43122)
0812 00:14:25.668245 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.668298 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 00:14:25.668509 8004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
I20260812 00:14:25.669003 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.668886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43126)
0812 00:14:25.668939 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.668970 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 00:14:25.669155 8056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:44087: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab.
Entry for principal kudu/127.16.99.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab.
I20260812 00:14:25.685736 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:44087
--webserver_interface=127.16.99.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab
--principal=kudu/127.16.99.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:45273,127.16.99.253:33897,127.16.99.252:44087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:25.689750 7978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } attempt: 3
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:14:25.697885 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.690260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.16.99.253:33897)
0812 00:14:25.690366 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.690369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.690379 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:25.690434 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.690435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.690450 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.690523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.690858 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.691425 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.692171 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.692414 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.692498 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.694910 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.694911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.694912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.694928 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.696612 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.697064 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.697065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.697066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.697129 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.697326 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.697328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.697328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.697452 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.697528 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.697697 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.697782 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 00:14:25.697885 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.690180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51486)
0812 00:14:25.690254 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.690264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.690334 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.690527 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.690528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.690696 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.690770 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.690772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.691529 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.691531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.692080 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.692082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.692496 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.692497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.692592 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.693207 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.693213 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.694797 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.696773 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.696774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.696775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.696974 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.697202 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.697203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.697203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.697270 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.697531 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.697532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.697589 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.697660 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.697779 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:14:25.698436 7978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:25.698803 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.698695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43128)
0812 00:14:25.698747 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.698776 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:14:25.699154 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.699066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43140)
0812 00:14:25.699107 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.699131 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 00:14:25.699254 7978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:44087: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
I20260812 00:14:25.718464 8056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } attempt: 1
I20260812 00:14:25.718994 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.718845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43148)
0812 00:14:25.718922 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.718963 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 00:14:25.719455 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.719346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43162)
0812 00:14:25.719401 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.719427 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 00:14:25.719579 8056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:44087: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
I20260812 00:14:25.730427 7978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } attempt: 1
I20260812 00:14:25.730957 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.730781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43166)
0812 00:14:25.730861 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.730916 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 00:14:25.731424 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.731309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43180)
0812 00:14:25.731362 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.731395 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 00:14:25.731571 7978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:44087: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
W20260812 00:14:25.772560 8063 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:14:25.772853 8063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:25.772892 8063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:25.772938 8063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:25.774290 8063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:25.774384 8063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:25.774411 8063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:25.774441 8063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:25.774470 8063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:25.776093 8063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:45273,127.16.99.253:33897,127.16.99.252:44087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab
--principal=kudu/127.16.99.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:44087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.99.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:25.776400 8063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:25.776647 8063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:25.781896 8069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:25.781893 8070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:25.781972 8072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:25.782276 8063 server_base.cc:1034] running on GCE node
I20260812 00:14:25.782581 8063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:25.782948 8063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:25.784108 8063 hybrid_clock.cc:648] HybridClock initialized: now 1786493665784086 us; error 39 us; skew 500 ppm
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:25.786927 8063 init.cc:377] Logged in from keytab as kudu/127.16.99.252@KRBTEST.COM (short username kudu)
I20260812 00:14:25.787406 8063 webserver.cc:460] Webserver started at http://127.16.99.252:45959/ using document root <none> and password file <none>
I20260812 00:14:25.787753 8063 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:25.787746 8056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } attempt: 2
I20260812 00:14:25.787837 8063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:25.788008 8063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:25.788213 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.788036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43194)
0812 00:14:25.788128 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.788172 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 00:14:25.788604 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.788497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43204)
0812 00:14:25.788547 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.788573 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:14:25.788693 8056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:44087: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
I20260812 00:14:25.789338 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data/instance:
uuid: "37850b94389c4abd99168397edcbca25"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "cfe85c08656833f1228ea576d9a504c5"
server_key_iv: "8030d27afaeafc29b1480f03f7de6a0d"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.789714 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/wal/instance:
uuid: "37850b94389c4abd99168397edcbca25"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "cfe85c08656833f1228ea576d9a504c5"
server_key_iv: "8030d27afaeafc29b1480f03f7de6a0d"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.791167 8063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260812 00:14:25.792068 8079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:25.792245 8063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 00:14:25.792352 8063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/wal
uuid: "37850b94389c4abd99168397edcbca25"
format_stamp: "Formatted at 2026-08-12 00:14:25 on dist-test-slave-zpfg"
server_key: "cfe85c08656833f1228ea576d9a504c5"
server_key_iv: "8030d27afaeafc29b1480f03f7de6a0d"
server_key_version: "encryptionkey@0"
I20260812 00:14:25.792429 8063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:25.806718 7978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } attempt: 2
I20260812 00:14:25.806802 8063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:25.807183 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.807014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43216)
0812 00:14:25.807081 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.807126 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:14:25.807557 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.807451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43226)
0812 00:14:25.807505 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.807535 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 00:14:25.807704 7978 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:44087: Network error: Client connection negotiation failed: client connection to 127.16.99.252:44087: connect: Connection refused (error 111)
I20260812 00:14:25.809079 8063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:25.809257 8063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:25.813740 8063 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.252:44087
I20260812 00:14:25.813776 8132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.252:44087 every 8 connection(s)
I20260812 00:14:25.814158 8063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
I20260812 00:14:25.815332 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:25.816979 8133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:25.822286 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 8063
I20260812 00:14:25.822439 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786493381608766-16783-0/minicluster-data/master-2/wal/instance
I20260812 00:14:25.822701 16783 external_mini_cluster.cc:1428] Setting key e5c276224f4219db08a48f5cf38f2eef
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:14:25.827728 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.817763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.16.99.254:45273)
0812 00:14:25.817827 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.817829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.818043 (+ 214us) server_negotiation.cc:407] Connection header received
0812 00:14:25.818622 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.818623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.818634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.818696 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.819377 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.819876 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.820880 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.821089 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.821150 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.824042 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.824044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.824045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.824054 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.825966 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.826627 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.826628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.826628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.826687 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.826961 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.826962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.826962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.827107 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.827171 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.827499 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.827604 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 00:14:25.827862 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.817728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:45273 (local address 127.0.0.1:58852)
0812 00:14:25.817917 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.817954 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.818521 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.818775 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.818778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.819026 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.819231 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.819235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.819958 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.819958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.820797 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.820800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.821201 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.821202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.821321 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.821853 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.821861 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.823937 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.826103 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.826113 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.826115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.826540 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.826746 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.826747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.826748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.826876 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.827171 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.827177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.827329 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.827447 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.827616 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260812 00:14:25.828519 8133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:14:25.833199 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.823170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.16.99.252:44087)
0812 00:14:25.823348 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.823353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.823558 (+ 205us) server_negotiation.cc:407] Connection header received
0812 00:14:25.823647 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.823651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.823680 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.823783 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.824412 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.825484 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.826530 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.827060 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.827126 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.830003 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.830013 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.830013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.830022 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.831924 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.832308 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.832312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.832312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.832377 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.832593 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.832593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.832594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.832790 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.832958 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.833002 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.833084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260812 00:14:25.836114 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.828844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:33897 (local address 127.0.0.1:51490)
0812 00:14:25.828902 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.828914 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.828958 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.829094 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.829094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.829260 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.829325 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.829327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.830018 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.830019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.830640 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.830641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.831166 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.831168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.831254 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.831645 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.831650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.832985 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.834948 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.834949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.834949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.835175 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.835391 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.835392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.835393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.835464 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.835866 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.835867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.835905 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.835961 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.836037 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 00:14:25.836452 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.828904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.16.99.253:33897)
0812 00:14:25.828950 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.828952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.828961 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:14:25.829024 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.829025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.829034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.829090 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.829405 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.829914 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.830727 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.831063 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.831156 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.833060 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.833062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.833062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.833070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.834846 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.835257 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.835258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.835258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.835323 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.835570 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.835572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.835572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.835712 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.835872 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.836011 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.836345 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 00:14:25.836879 8133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:14:25.842837 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.833979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.16.99.254:45273)
0812 00:14:25.834027 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.834029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.834039 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:25.834117 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.834119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.834130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.834191 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.834945 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.835692 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.836746 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.837019 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.837094 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.839763 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.839765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.839765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.839773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.841356 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.841854 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.841855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.841856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.841923 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.842174 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.842175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.842175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.842329 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.842547 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.842592 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.842672 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 00:14:25.844949 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.837176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43252)
0812 00:14:25.837218 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.837227 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.837266 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.837474 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.837475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.837631 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.837673 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.837674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.838697 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.838698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.839485 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.839488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.839817 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.839818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.839879 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.840314 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.840318 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.841670 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.843707 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.843708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.843708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.843877 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.844108 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.844109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.844109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.844361 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.844603 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.844604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.844648 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.844718 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.844817 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 00:14:25.844949 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.837217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.16.99.252:44087)
0812 00:14:25.837260 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.837262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.837270 (+ 8us) server_negotiation.cc:407] Connection header received
0812 00:14:25.837398 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.837398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.837408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.837458 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.837749 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.838593 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.839566 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.839757 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.839817 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.841859 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.841860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.841861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.841867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.843631 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.843953 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.843955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.843955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.844015 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.844434 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.844435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.844435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.844537 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.844587 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.844746 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.844860 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":24}
I20260812 00:14:25.846788 8133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: Bootstrap starting.
I20260812 00:14:25.847908 8133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:25.848256 8133 log.cc:824] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: Log is configured to *not* fsync() on all Append() calls
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:14:25.849221 8133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: No bootstrap required, opened a new log
I20260812 00:14:25.850915 8133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.851074 8133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37850b94389c4abd99168397edcbca25, State: Initialized, Role: FOLLOWER
I20260812 00:14:25.851184 8133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.851577 8133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:25.851902 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.843466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.16.99.253:33897)
0812 00:14:25.843512 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.843513 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.843520 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:14:25.843540 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.843541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.843549 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.843594 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.844266 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.845011 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.846101 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.846419 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.846498 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.848961 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.848963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.848963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.848972 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.850632 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.850977 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.850978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.850978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.851037 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.851233 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.851234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.851235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.851382 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.851529 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.851717 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.851813 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 00:14:25.851526 8139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.853210 8139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:25.853672 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 00:14:25.853804 8150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:25.853842 8150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:25.854996 7949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9a8807723d454611b2d1e26bbffa9343"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58866
I20260812 00:14:25.854025 8063 master_runner.cc:186] Error getting master registration for 127.16.99.254:45273: OK, instance_id { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" instance_seqno: 1786493665459682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:14:25.856531 8063 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:14:25.879874 8056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } attempt: 3
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:14:25.888836 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.880314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43260)
0812 00:14:25.880376 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.880390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.880442 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.880619 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.880620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.880832 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.880891 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.880893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.881917 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.881918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.882794 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.882797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.883334 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.883335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.883398 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.883983 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.883989 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.885601 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.887588 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.887589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.887589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.887840 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.888059 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.888060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.888060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.888124 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.888487 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.888488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.888535 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.888604 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.888721 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:14:25.888836 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.880395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.16.99.252:44087)
0812 00:14:25.880477 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.880479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.880491 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:14:25.880519 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.880523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.880537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.880599 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.880997 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.881789 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.882905 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.883234 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.883323 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.885710 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.885712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.885712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.885721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.887487 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.887921 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.887922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.887922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.887986 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.888189 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.888189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.888190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.888415 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.888490 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.888646 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.888735 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 00:14:25.891017 8056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: Bootstrap starting.
I20260812 00:14:25.891812 8056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:25.892145 8056 log.cc:824] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:25.893251 8056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: No bootstrap required, opened a new log
I20260812 00:14:25.895493 8056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.895660 8056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42991915533349f08da28c3c573a0aef, State: Initialized, Role: FOLLOWER
I20260812 00:14:25.895744 8056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.896157 8056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:25.896116 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.896225 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [sys.catalog]: This master's current role is: FOLLOWER
W20260812 00:14:25.898258 8164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:25.898476 8164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:25.898520 7985 master_runner.cc:186] Error getting master registration for 127.16.99.254:45273: OK, instance_id { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" instance_seqno: 1786493665459682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:14:25.900965 7985 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:14:25.906906 7978 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } attempt: 3
Aug 12 00:14:25 dist-test-slave-zpfg krb5kdc[7891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493665, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:14:25.917838 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.907464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.16.99.252:44087)
0812 00:14:25.907572 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.907575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.907600 (+ 25us) server_negotiation.cc:407] Connection header received
0812 00:14:25.907628 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.907629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.907639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.907688 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.908121 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.908948 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.910068 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.910411 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.910519 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.913141 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.913143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.913144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.913154 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.916322 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.916823 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.916825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.916825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.916949 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.917209 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.917211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.917211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.917377 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.917446 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.917653 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.917736 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 00:14:25.917852 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.907319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:44087 (local address 127.0.0.1:43270)
0812 00:14:25.907390 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:25.907406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:25.907472 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:25.907721 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:25.907723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:25.907943 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:25.908006 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.908008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.909036 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.909037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.909958 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:25.909961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.910539 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.910540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:25.910620 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.911281 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:25.911287 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:25.913018 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:25.916456 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.916458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.916459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.916718 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.917040 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:25.917041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:25.917041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:25.917119 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:25.917454 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:25.917455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:25.917530 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 00:14:25.917613 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 00:14:25.917735 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 00:14:25.920015 7978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343: Bootstrap starting.
I20260812 00:14:25.920888 7978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:25.921350 7978 log.cc:824] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:25.922405 7978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343: No bootstrap required, opened a new log
I20260812 00:14:25.924077 7978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.924350 7978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8807723d454611b2d1e26bbffa9343, State: Initialized, Role: FOLLOWER
I20260812 00:14:25.924537 7978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.925052 7949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 00:14:25.925009 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.925140 7949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:25.925143 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:25.926060 7949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.926704 8107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a8807723d454611b2d1e26bbffa9343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "37850b94389c4abd99168397edcbca25"
I20260812 00:14:25.926744 8030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a8807723d454611b2d1e26bbffa9343" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "42991915533349f08da28c3c573a0aef"
I20260812 00:14:25.926847 8107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:25.926900 8030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:25.927773 8107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8807723d454611b2d1e26bbffa9343 in term 1.
I20260812 00:14:25.928102 8030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8807723d454611b2d1e26bbffa9343 in term 1.
I20260812 00:14:25.928086 7922 leader_election.cc:302] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37850b94389c4abd99168397edcbca25, 9a8807723d454611b2d1e26bbffa9343; no voters:
I20260812 00:14:25.928489 7949 leader_election.cc:288] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [CANDIDATE]: Term 1 election: Requested vote from peers 42991915533349f08da28c3c573a0aef (127.16.99.253:33897), 37850b94389c4abd99168397edcbca25 (127.16.99.252:44087)
I20260812 00:14:25.925025 7978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:25.930454 8167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:25.931097 8167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [term 1 LEADER]: Becoming Leader. State: Replica: 9a8807723d454611b2d1e26bbffa9343, State: Running, Role: LEADER
I20260812 00:14:25.931227 8167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } }
I20260812 00:14:25.931847 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8807723d454611b2d1e26bbffa9343. Latest consensus state: current_term: 1 leader_uuid: "9a8807723d454611b2d1e26bbffa9343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.931941 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: This master's current role is: LEADER
W20260812 00:14:25.935720 8183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:25.935874 8183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:25.938515 8187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 00:14:25.938779 8187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:25.939699 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.929933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.16.99.252:44087)
0812 00:14:25.929981 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.929983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.930439 (+ 456us) server_negotiation.cc:407] Connection header received
0812 00:14:25.930546 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.930547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.930559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.930588 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.932019 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.932719 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.933934 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.934204 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.934260 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.936052 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.936053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.936054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.936080 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.937910 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.938429 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.938430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.938431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.938503 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.938999 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.939000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.939000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.939155 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.939256 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.939487 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.939578 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:25.946102 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.931608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.16.99.254:45273)
0812 00:14:25.931654 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.931656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.936188 (+ 4532us) server_negotiation.cc:407] Connection header received
0812 00:14:25.936309 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.936310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.936322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.936666 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.937633 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.938331 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.940458 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.940748 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.940797 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.942629 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.942631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.942631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.942639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.944724 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.945158 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.945160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.945160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.945234 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.945467 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.945468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.945468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.945609 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.945686 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.945911 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.945998 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:25.947463 8107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [term 1 FOLLOWER]: Refusing update from remote peer 9a8807723d454611b2d1e26bbffa9343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:25.947649 8030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [term 1 FOLLOWER]: Refusing update from remote peer 9a8807723d454611b2d1e26bbffa9343: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:25.948235 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:25.931901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.16.99.253:33897)
0812 00:14:25.931955 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:25.931956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:25.939533 (+ 7577us) server_negotiation.cc:407] Connection header received
0812 00:14:25.939641 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:25.939642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:25.939654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:25.939684 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:25.941246 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.941946 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.942980 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:25.943285 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:25.943330 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:25.944787 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:25.944788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:25.944788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:25.944797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:25.946564 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.947087 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.947089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.947089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.947150 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:25.947483 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:25.947483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:25.947484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:25.947611 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:25.947678 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:25.948038 (+ 360us) server_negotiation.cc:299] Negotiation successful
0812 00:14:25.948132 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:25.948299 8167 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:25.948463 8184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:25.949965 8139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8807723d454611b2d1e26bbffa9343. Latest consensus state: current_term: 1 leader_uuid: "9a8807723d454611b2d1e26bbffa9343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.950124 8139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:25.950474 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8807723d454611b2d1e26bbffa9343. Latest consensus state: current_term: 1 leader_uuid: "9a8807723d454611b2d1e26bbffa9343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.950567 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:25.951833 8191 mvcc.cc:205] Tried to move back new op lower bound from 7317478055717556224 to 7317478055655469056. Current Snapshot: MvccSnapshot[applied={T|T < 7317478055717556224}]
I20260812 00:14:25.951949 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a8807723d454611b2d1e26bbffa9343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.952019 8192 mvcc.cc:205] Tried to move back new op lower bound from 7317478055717556224 to 7317478055655469056. Current Snapshot: MvccSnapshot[applied={T|T < 7317478055717556224}]
I20260812 00:14:25.952064 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:25.952121 8139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a8807723d454611b2d1e26bbffa9343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.952201 8139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:25.952463 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a8807723d454611b2d1e26bbffa9343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.952565 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:25.952966 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a8807723d454611b2d1e26bbffa9343" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8807723d454611b2d1e26bbffa9343" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 45273 } } peers { permanent_uuid: "42991915533349f08da28c3c573a0aef" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 33897 } } peers { permanent_uuid: "37850b94389c4abd99168397edcbca25" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 44087 } } }
I20260812 00:14:25.953084 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:25.955159 8187 catalog_manager.cc:1347] Generated new cluster ID: f7c7a28defcd48c8a0fac01c7c79a1ed
I20260812 00:14:25.955222 8187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:25.984752 8187 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:25.985626 8187 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:25.999039 8187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a8807723d454611b2d1e26bbffa9343: Generated new TSK 0
I20260812 00:14:25.999302 8187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 00:14:26.026062 7981 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.024476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.16.99.254:45273)
0812 00:14:26.024537 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.024539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.025283 (+ 744us) server_negotiation.cc:407] Connection header received
0812 00:14:26.025401 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.025402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.025414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.025447 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.025962 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58882: BlockingRecv error: recv got EOF from 127.0.0.1:58882 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260812 00:14:26.026957 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.024964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.16.99.253:33897)
0812 00:14:26.025008 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.025009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.026396 (+ 1387us) server_negotiation.cc:407] Connection header received
0812 00:14:26.026502 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.026504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.026515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.026545 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.026890 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51512: BlockingRecv error: recv got EOF from 127.0.0.1:51512 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:14:26.028244 8137 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.027378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.16.99.252:44087)
0812 00:14:26.027424 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.027426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.027610 (+ 184us) server_negotiation.cc:407] Connection header received
0812 00:14:26.027706 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.027707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.027718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.027749 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.028157 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43290: BlockingRecv error: recv got EOF from 127.0.0.1:43290 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260812 00:14:26.030953 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.030056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.16.99.253:33897)
0812 00:14:26.030096 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.030098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.030303 (+ 205us) server_negotiation.cc:407] Connection header received
0812 00:14:26.030474 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.030475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.030497 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.030527 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.030898 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51520: BlockingRecv error: recv got EOF from 127.0.0.1:51520 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 00:14:26.034463 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.033484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.16.99.253:33897)
0812 00:14:26.033521 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.033523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.033859 (+ 336us) server_negotiation.cc:407] Connection header received
0812 00:14:26.033969 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.033971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.033982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.034012 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.034411 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51534: BlockingRecv error: recv got EOF from 127.0.0.1:51534 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 00:14:26.046253 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.045153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.16.99.253:33897)
0812 00:14:26.045204 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.045206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.045519 (+ 313us) server_negotiation.cc:407] Connection header received
0812 00:14:26.045642 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.045643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.045656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.045689 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.046188 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51536: BlockingRecv error: recv got EOF from 127.0.0.1:51536 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:14:26.052762 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.051899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.16.99.253:33897)
0812 00:14:26.051945 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.051947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.052127 (+ 180us) server_negotiation.cc:407] Connection header received
0812 00:14:26.052163 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.052164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.052177 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.052259 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.052693 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51542: BlockingRecv error: recv got EOF from 127.0.0.1:51542 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 00:14:26.063408 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.062414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.16.99.253:33897)
0812 00:14:26.062468 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.062470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.062642 (+ 172us) server_negotiation.cc:407] Connection header received
0812 00:14:26.062722 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.062723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.062758 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.062831 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.063344 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51556: BlockingRecv error: recv got EOF from 127.0.0.1:51556 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260812 00:14:26.081799 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.080732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.16.99.253:33897)
0812 00:14:26.080783 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.080786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.081138 (+ 352us) server_negotiation.cc:407] Connection header received
0812 00:14:26.081209 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.081210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.081224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.081299 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.081735 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51562: BlockingRecv error: recv got EOF from 127.0.0.1:51562 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 00:14:26.115849 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.114941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.16.99.253:33897)
0812 00:14:26.115035 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.115038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.115271 (+ 233us) server_negotiation.cc:407] Connection header received
0812 00:14:26.115301 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.115302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.115313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.115361 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.115775 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51576: BlockingRecv error: recv got EOF from 127.0.0.1:51576 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 00:14:26.182185 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.181401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.16.99.253:33897)
0812 00:14:26.181474 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.181480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.181516 (+ 36us) server_negotiation.cc:407] Connection header received
0812 00:14:26.181544 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.181545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.181556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.181601 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.182116 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51584: BlockingRecv error: recv got EOF from 127.0.0.1:51584 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 00:14:26.312439 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.311665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.16.99.253:33897)
0812 00:14:26.311740 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.311742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.311837 (+ 95us) server_negotiation.cc:407] Connection header received
0812 00:14:26.311882 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.311883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.311894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.311959 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.312374 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51590: BlockingRecv error: recv got EOF from 127.0.0.1:51590 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260812 00:14:26.570750 8060 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:26.569969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.16.99.253:33897)
0812 00:14:26.570046 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:26.570048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:26.570059 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:14:26.570086 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:26.570087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:26.570097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:26.570165 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:26.570698 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51604: BlockingRecv error: recv got EOF from 127.0.0.1:51604 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:14:26.854516 8150 catalog_manager.cc:1259] Loaded cluster ID: f7c7a28defcd48c8a0fac01c7c79a1ed
I20260812 00:14:26.854611 8150 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: loading cluster ID for follower catalog manager: success
I20260812 00:14:26.855499 8150 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: acquiring CA information for follower catalog manager: success
I20260812 00:14:26.855839 8150 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 37850b94389c4abd99168397edcbca25: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:14:26.899065 8164 catalog_manager.cc:1259] Loaded cluster ID: f7c7a28defcd48c8a0fac01c7c79a1ed
I20260812 00:14:26.899151 8164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: loading cluster ID for follower catalog manager: success
I20260812 00:14:26.900084 8164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: acquiring CA information for follower catalog manager: success
I20260812 00:14:26.900445 8164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42991915533349f08da28c3c573a0aef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:14:27.087025 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.084142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.16.99.253:33897)
0812 00:14:27.084255 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:27.084257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:27.084271 (+ 14us) server_negotiation.cc:407] Connection header received
0812 00:14:27.084303 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:27.084303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:27.084354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:27.084444 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:27.084684 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.085199 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.086095 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.086313 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.086388 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:27.086620 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:27.086735 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:27.086802 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:27.086912 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 00:14:27.086938 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260812 00:14:27.091686 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.089089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.16.99.252:44087)
0812 00:14:27.089208 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:27.089210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:27.089243 (+ 33us) server_negotiation.cc:407] Connection header received
0812 00:14:27.089284 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:27.089284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:27.089321 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:27.089385 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:27.089481 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.089998 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.090850 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.091099 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.091180 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:27.091376 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:27.091495 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:27.091573 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:27.091592 (+ 19us) server_negotiation.cc:299] Negotiation successful
0812 00:14:27.091612 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260812 00:14:27.096472 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.093696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43306 (local address 127.16.99.252:44087)
0812 00:14:27.093738 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:27.093740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:27.093858 (+ 118us) server_negotiation.cc:407] Connection header received
0812 00:14:27.093960 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:27.093961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:27.093971 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:27.094003 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:27.094190 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.094798 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.095655 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.095919 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.095996 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:27.096171 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:27.096203 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:27.096250 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:27.096334 (+ 84us) server_negotiation.cc:299] Negotiation successful
0812 00:14:27.096360 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:14:27.097271 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.093947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.16.99.253:33897)
0812 00:14:27.093994 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:27.093996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:27.094008 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:14:27.094116 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:27.094118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:27.094133 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:27.094183 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:27.094455 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.095077 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.096234 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.096529 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.096611 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:27.096952 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:27.096989 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:27.097041 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:27.097166 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 00:14:27.097193 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 00:14:27.097858 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:27.093623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.16.99.254:45273)
0812 00:14:27.093851 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:27.093854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:27.093865 (+ 11us) server_negotiation.cc:407] Connection header received
0812 00:14:27.093915 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:27.093916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:27.093975 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:27.094058 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:27.094784 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.095523 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.096579 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:27.097072 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:27.097131 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:27.097480 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:27.097582 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:27.097667 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:27.097706 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 00:14:27.097733 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
I20260812 00:14:27.100051 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7904
I20260812 00:14:27.108256 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 7985
I20260812 00:14:27.114934 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 8063
2026-08-12T00:14:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3067 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 00:14:27 dist-test-slave-zpfg krb5kdc[8283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 00:14:27 dist-test-slave-zpfg krb5kdc[8283](info): set up 2 sockets
Aug 12 00:14:27 dist-test-slave-zpfg krb5kdc[8283](info): commencing operation
krb5kdc: starting...
W20260812 00:14:28.292326 16783 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/test-user.keytab.
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T00:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T00:14:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.99.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
Entry for principal kudu/127.16.99.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab.
I20260812 00:14:28.434557 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:34891
--webserver_interface=127.16.99.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:34891,127.16.99.253:38051,127.16.99.252:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 00:14:28.519446 8296 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:14:28.519765 8296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:28.519820 8296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:28.519861 8296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:28.521175 8296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:28.521255 8296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:28.521277 8296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:28.521301 8296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:28.521328 8296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:28.522922 8296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:34891,127.16.99.253:38051,127.16.99.252:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.254.keytab
--principal=kudu/127.16.99.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.254:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.99.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:28.523214 8296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:28.523463 8296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:28.528751 8301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:28.528841 8302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:28.528751 8304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:28.529050 8296 server_base.cc:1034] running on GCE node
I20260812 00:14:28.529394 8296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:28.529666 8296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:28.530840 8296 hybrid_clock.cc:648] HybridClock initialized: now 1786493668530833 us; error 41 us; skew 500 ppm
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:28.541237 8296 init.cc:377] Logged in from keytab as kudu/127.16.99.254@KRBTEST.COM (short username kudu)
I20260812 00:14:28.541838 8296 webserver.cc:460] Webserver started at http://127.16.99.254:39017/ using document root <none> and password file <none>
I20260812 00:14:28.542106 8296 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:28.542177 8296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:28.542335 8296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:28.543550 8296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data/instance:
uuid: "81832955c5844feba40197b88c2f9511"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "3f95bd48dec06144cf8bb8f126a4cabb"
server_key_iv: "dac9ba2090a1fce1fb0be226ad4b3264"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.543982 8296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/wal/instance:
uuid: "81832955c5844feba40197b88c2f9511"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "3f95bd48dec06144cf8bb8f126a4cabb"
server_key_iv: "dac9ba2090a1fce1fb0be226ad4b3264"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.545439 8296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:28.546268 8311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:28.546468 8296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:28.546590 8296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/wal
uuid: "81832955c5844feba40197b88c2f9511"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "3f95bd48dec06144cf8bb8f126a4cabb"
server_key_iv: "dac9ba2090a1fce1fb0be226ad4b3264"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.546669 8296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:28.573417 8296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:28.576081 8296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:28.576274 8296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:28.580670 8296 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.254:34891
I20260812 00:14:28.580710 8366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.254:34891 every 8 connection(s)
I20260812 00:14:28.581125 8296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/data/info.pb
I20260812 00:14:28.581286 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 8296
I20260812 00:14:28.581396 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-0/wal/instance
I20260812 00:14:28.581621 16783 external_mini_cluster.cc:1428] Setting key 15bf9762f4ea4b6ee5a192db0c8ee091
I20260812 00:14:28.582302 8367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:28.583938 8367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.99.253@KRBTEST.COM; defaulting to no policy
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.253@KRBTEST.COM" created.
I20260812 00:14:28.595592 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.584707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:34891 (local address 127.0.0.1:35512)
0812 00:14:28.584912 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.584952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.585682 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.586012 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.586015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.586443 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.586706 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.586713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.587835 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.587837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.589042 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.589045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.589588 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.589589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.589672 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.590230 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.590241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.592038 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.593896 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.593903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.593905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.594140 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.594451 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.594453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.594453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.594552 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.594942 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.594948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.595013 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.595093 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.595303 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":125,"threads_started":1}
I20260812 00:14:28.595594 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.584734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.16.99.254:34891)
0812 00:14:28.584944 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.584952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.585048 (+ 96us) server_negotiation.cc:407] Connection header received
0812 00:14:28.585863 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.585865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.585883 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.585974 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.586839 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.587694 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.589134 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.589468 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.589553 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.592158 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.592169 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.592169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.592179 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.593811 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.594226 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.594229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.594229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.594347 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.594615 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.594615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.594616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.594873 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.594940 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.595170 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.595370 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":164,"thread_start_us":131,"threads_started":1}
I20260812 00:14:28.596529 8367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:28.596951 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.596823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35648)
0812 00:14:28.596877 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.596929 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 00:14:28.597127 8315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
I20260812 00:14:28.597635 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.597518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35654)
0812 00:14:28.597574 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.597608 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 00:14:28.597805 8367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:38051: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab.
Entry for principal kudu/127.16.99.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab.
I20260812 00:14:28.614053 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:38051
--webserver_interface=127.16.99.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab
--principal=kudu/127.16.99.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:34891,127.16.99.253:38051,127.16.99.252:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:28.647073 8367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } attempt: 1
I20260812 00:14:28.647673 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.647504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35668)
0812 00:14:28.647599 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.647637 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 00:14:28.648097 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.647967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35674)
0812 00:14:28.648024 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.648068 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 00:14:28.648228 8367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:38051: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
W20260812 00:14:28.698478 8374 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:14:28.698724 8374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:28.698742 8374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:28.698766 8374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:28.700026 8374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:28.700075 8374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:28.700090 8374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:28.700102 8374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:28.700115 8374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:28.701704 8374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:34891,127.16.99.253:38051,127.16.99.252:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.253.keytab
--principal=kudu/127.16.99.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.253:38051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.99.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:28.701987 8374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:28.702268 8374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:28.707837 8379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:28.707845 8380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:28.707877 8374 server_base.cc:1034] running on GCE node
W20260812 00:14:28.707988 8382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:28.708436 8374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:28.708756 8374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:28.709913 8374 hybrid_clock.cc:648] HybridClock initialized: now 1786493668709902 us; error 49 us; skew 500 ppm
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:28.712580 8374 init.cc:377] Logged in from keytab as kudu/127.16.99.253@KRBTEST.COM (short username kudu)
I20260812 00:14:28.713047 8374 webserver.cc:460] Webserver started at http://127.16.99.253:36343/ using document root <none> and password file <none>
I20260812 00:14:28.713294 8374 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:28.713364 8374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:28.713517 8374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:28.714685 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data/instance:
uuid: "6ceb183b90f5458190d12681084bc595"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "1349a39640275df61ecf4f9a2bde5f0d"
server_key_iv: "de279f92ea4982ca2313a46ab75210e2"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.715052 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/wal/instance:
uuid: "6ceb183b90f5458190d12681084bc595"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "1349a39640275df61ecf4f9a2bde5f0d"
server_key_iv: "de279f92ea4982ca2313a46ab75210e2"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.716369 8367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } attempt: 2
I20260812 00:14:28.716529 8374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 00:14:28.716816 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.716664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35684)
0812 00:14:28.716731 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.716775 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 00:14:28.717193 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.717098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35696)
0812 00:14:28.717147 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.717173 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 00:14:28.717289 8367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.253:38051: Network error: Client connection negotiation failed: client connection to 127.16.99.253:38051: connect: Connection refused (error 111)
I20260812 00:14:28.717555 8389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:28.717731 8374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 00:14:28.717851 8374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/wal
uuid: "6ceb183b90f5458190d12681084bc595"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "1349a39640275df61ecf4f9a2bde5f0d"
server_key_iv: "de279f92ea4982ca2313a46ab75210e2"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.717950 8374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:28.733865 8374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:28.736380 8374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:28.736506 8374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:28.740792 8374 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.253:38051
I20260812 00:14:28.740854 8443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.253:38051 every 8 connection(s)
I20260812 00:14:28.741227 8374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/data/info.pb
I20260812 00:14:28.742322 8444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:28.743940 8444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:28.750105 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 8374
I20260812 00:14:28.750226 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-1/wal/instance
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:14:28.750460 16783 external_mini_cluster.cc:1428] Setting key 396389bc6a0d77dc34e565b001f47527
I20260812 00:14:28.754547 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.744677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.16.99.254:34891)
0812 00:14:28.744738 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.744741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.744934 (+ 193us) server_negotiation.cc:407] Connection header received
0812 00:14:28.745464 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.745466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.745484 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.745548 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.746186 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.746832 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.747890 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.748110 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.748199 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.750832 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.750834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.750834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.750842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.752672 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.753223 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.753225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.753225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.753296 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.753630 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.753631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.753631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.753791 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.753866 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.754261 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.754402 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 00:14:28.754704 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.744622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:34891 (local address 127.0.0.1:35524)
0812 00:14:28.744808 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.744844 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.745351 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.745625 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.745629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.745889 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.746086 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.746090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.746929 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.746931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.747794 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.747796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.748222 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.748224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.748363 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.748909 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.748918 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.750723 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.752776 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.752789 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.752792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.753037 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.753418 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.753419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.753419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.753545 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.753871 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.753878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.754022 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.754216 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.754461 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260812 00:14:28.755563 8444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.99.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.99.252@KRBTEST.COM" created.
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:14:28.766741 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.755869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35708)
0812 00:14:28.755947 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.755960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.756017 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.756372 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.756373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.756522 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.756576 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.756578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.757517 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.757518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.758029 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.758030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.758596 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.758597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.758664 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.759147 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.759152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.763413 (+ 4261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.765550 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.765551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.765552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.765791 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.766019 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.766020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.766021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.766091 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.766487 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.766488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.766528 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.766583 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.766668 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 00:14:28.766899 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.756006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.16.99.253:38051)
0812 00:14:28.756216 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.756220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.756244 (+ 24us) server_negotiation.cc:407] Connection header received
0812 00:14:28.756261 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.756263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.756274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.756359 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.756662 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.757409 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.758111 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.758514 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.758601 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.763527 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.763544 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.763545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.763555 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.765447 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.765888 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.765891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.765891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.765949 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.766165 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.766166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.766166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.766417 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.766512 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.766642 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.766746 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
I20260812 00:14:28.767557 8444 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:28.767975 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.767844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57570)
0812 00:14:28.767895 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.767944 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 00:14:28.768136 8391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
I20260812 00:14:28.768613 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.768522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57578)
0812 00:14:28.768570 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.768596 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 00:14:28.768893 8444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:40389: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.99.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab.
Entry for principal kudu/127.16.99.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab.
I20260812 00:14:28.781845 16783 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcVEJCj/build/release/bin/kudu
/tmp/dist-test-taskcVEJCj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:40389
--webserver_interface=127.16.99.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab
--principal=kudu/127.16.99.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.99.212:36569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.99.254:34891,127.16.99.253:38051,127.16.99.252:40389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 00:14:28.808454 8367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } attempt: 3
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:14:28.815588 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.808922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.16.99.253:38051)
0812 00:14:28.808998 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.809000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.809012 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:14:28.809041 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.809046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.809058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.809122 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.809480 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.809983 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.810774 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.811006 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.811086 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.813003 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.813004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.813004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.813012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.814411 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.814773 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.814775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.814775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.814836 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.815036 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.815037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.815037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.815156 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.815235 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.815411 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.815490 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 00:14:28.815589 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.808824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35716)
0812 00:14:28.808886 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.808898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.808968 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.809127 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.809128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.809307 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.809384 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.809386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.810084 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.810085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.810688 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.810691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.811084 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.811085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.811168 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.811674 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.811679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.812901 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.814504 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.814505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.814506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.814684 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.814902 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.814903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.814904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.814964 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.815238 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.815239 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.815294 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.815375 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.815484 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 00:14:28.816004 8367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:28.816403 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.816289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57594)
0812 00:14:28.816342 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.816372 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:14:28.816812 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.816713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57596)
0812 00:14:28.816765 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.816795 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 00:14:28.816918 8367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:40389: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
I20260812 00:14:28.818123 8444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } attempt: 1
I20260812 00:14:28.818503 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.818388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57598)
0812 00:14:28.818432 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.818470 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260812 00:14:28.818856 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.818755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57610)
0812 00:14:28.818809 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.818837 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 00:14:28.818986 8444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:40389: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
I20260812 00:14:28.848057 8367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } attempt: 1
I20260812 00:14:28.848574 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.848388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57612)
0812 00:14:28.848466 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.848524 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 00:14:28.849072 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.848929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57624)
0812 00:14:28.848984 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.849024 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 00:14:28.849193 8367 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:40389: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
W20260812 00:14:28.868623 8451 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 00:14:28.868892 8451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 00:14:28.868924 8451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 00:14:28.868970 8451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 00:14:28.870127 8451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 00:14:28.870196 8451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 00:14:28.870219 8451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 00:14:28.870239 8451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 00:14:28.870266 8451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 00:14:28.871881 8451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.99.212:36569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.99.254:34891,127.16.99.253:38051,127.16.99.252:40389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/krb5kdc/kudu_127.16.99.252.keytab
--principal=kudu/127.16.99.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.99.252:40389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.99.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 11 Aug 2026 22:57:27 UTC on ccycloud.quasar-xbduhr.root.comops.site
build id 4581601
I20260812 00:14:28.872184 8451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 00:14:28.872433 8451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 00:14:28.877978 8458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:28.878074 8451 server_base.cc:1034] running on GCE node
W20260812 00:14:28.878171 8460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 00:14:28.877941 8457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 00:14:28.878540 8451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 00:14:28.878877 8451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 00:14:28.880052 8451 hybrid_clock.cc:648] HybridClock initialized: now 1786493668880041 us; error 48 us; skew 500 ppm
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 00:14:28.882632 8451 init.cc:377] Logged in from keytab as kudu/127.16.99.252@KRBTEST.COM (short username kudu)
I20260812 00:14:28.883126 8451 webserver.cc:460] Webserver started at http://127.16.99.252:38243/ using document root <none> and password file <none>
I20260812 00:14:28.883381 8451 fs_manager.cc:359] Metadata directory not provided
I20260812 00:14:28.883431 8451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 00:14:28.883574 8451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 00:14:28.884774 8451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data/instance:
uuid: "e8cc029345b2417d9e10b041f83227d9"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "b53a05995a8b827082e91a2d6c9f1821"
server_key_iv: "f4f738548b0610feb063d3ce1941dd54"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.885139 8451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/wal/instance:
uuid: "e8cc029345b2417d9e10b041f83227d9"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "b53a05995a8b827082e91a2d6c9f1821"
server_key_iv: "f4f738548b0610feb063d3ce1941dd54"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.886619 8451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 00:14:28.887136 8444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } attempt: 2
I20260812 00:14:28.887611 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.887471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57626)
0812 00:14:28.887527 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.887564 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 00:14:28.887691 8468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 00:14:28.888031 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.887918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57640)
0812 00:14:28.887968 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.887999 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 00:14:28.888036 8451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
W20260812 00:14:28.888125 8444 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.99.252:40389: Network error: Client connection negotiation failed: client connection to 127.16.99.252:40389: connect: Connection refused (error 111)
I20260812 00:14:28.888149 8451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data,/tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/wal
uuid: "e8cc029345b2417d9e10b041f83227d9"
format_stamp: "Formatted at 2026-08-12 00:14:28 on dist-test-slave-zpfg"
server_key: "b53a05995a8b827082e91a2d6c9f1821"
server_key_iv: "f4f738548b0610feb063d3ce1941dd54"
server_key_version: "encryptionkey@0"
I20260812 00:14:28.888257 8451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 00:14:28.908118 8451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 00:14:28.910791 8451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 00:14:28.910997 8451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 00:14:28.915781 8451 rpc_server.cc:307] RPC server started. Bound to: 127.16.99.252:40389
I20260812 00:14:28.915820 8521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.99.252:40389 every 8 connection(s)
I20260812 00:14:28.916275 8451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/data/info.pb
I20260812 00:14:28.917450 8522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 00:14:28.917995 16783 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcVEJCj/build/release/bin/kudu as pid 8451
I20260812 00:14:28.918094 16783 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcVEJCj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786493381608766-16783-0/minicluster-data/master-2/wal/instance
I20260812 00:14:28.918401 16783 external_mini_cluster.cc:1428] Setting key 9f102fb370a1a85aa8c3300746b5320b
I20260812 00:14:28.920149 8522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:28.924353 8367 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } attempt: 2
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.254@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:14:28.931883 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.921029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.254:34891 (local address 127.0.0.1:35538)
0812 00:14:28.921335 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.921385 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.921478 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.921711 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.921714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.922128 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.922273 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.922280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.923000 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.923001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.923866 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.923868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.924293 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.924295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.924375 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.925307 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.925317 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.927591 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.930100 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.930109 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.930112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.930528 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.930886 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.930887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.930887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.930965 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.931384 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.931391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.931452 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.931538 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.931715 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":27,"thread_start_us":232,"threads_started":1}
I20260812 00:14:28.931943 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.921079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.16.99.254:34891)
0812 00:14:28.921128 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.921202 (+ 74us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.921485 (+ 283us) server_negotiation.cc:407] Connection header received
0812 00:14:28.921540 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.921541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.921559 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.921662 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.922406 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.922902 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.923956 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.924171 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.924241 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.927713 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.927715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.927715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.927724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.930023 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.930635 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.930636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.930636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.930790 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.931038 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.931040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.931040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.931295 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.931376 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.931728 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.931826 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:28.931883 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.919096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.16.99.252:40389)
0812 00:14:28.919369 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.919379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.919443 (+ 64us) server_negotiation.cc:407] Connection header received
0812 00:14:28.920044 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.920054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.920122 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.920317 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.921300 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.922806 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.923601 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.924093 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.924188 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.927630 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.927642 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.927642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.927651 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.929829 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.930310 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.930313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.930313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.930461 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.930706 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.930707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.930707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.931087 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.931379 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.931458 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.931617 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
I20260812 00:14:28.932519 8522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } has no permanent_uuid. Determining permanent_uuid...
I20260812 00:14:28.934504 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.924643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57662)
0812 00:14:28.924691 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.924700 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.924761 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.925049 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.925051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.925270 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.925320 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.925322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.926015 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.926016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.926732 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.926734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.927254 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.927255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.927320 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.927836 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.927841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.929103 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.933213 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.933214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.933215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.933475 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.933722 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.933723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.933723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.933800 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.934119 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.934120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.934182 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.934232 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.934376 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 00:14:28.934981 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.924702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.16.99.252:40389)
0812 00:14:28.924878 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.924880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.924889 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:14:28.924910 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.924911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.924921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.925031 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.925395 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.925906 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.926856 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.927123 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.927254 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.929426 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.929427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.929428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.929435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.933085 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.933570 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.933571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.933572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.933640 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.933885 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.933886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.933887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.934026 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.934698 (+ 672us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.934741 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.934826 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260812 00:14:28.937239 8367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511: Bootstrap starting.
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:14:28.938127 8367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511: Neither blocks nor log segments found. Creating new log.
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.254@KRBTEST.COM
I20260812 00:14:28.938555 8367 log.cc:824] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:28.939848 8367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511: No bootstrap required, opened a new log
I20260812 00:14:28.942039 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.932869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.16.99.253:38051)
0812 00:14:28.932913 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.932915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.933116 (+ 201us) server_negotiation.cc:407] Connection header received
0812 00:14:28.933151 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.933152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.933167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.933227 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.933611 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.934428 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.935501 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.935841 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.935890 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.938840 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.938841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.938842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.938850 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.940715 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.941138 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.941139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.941140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.941199 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.941423 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.941423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.941424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.941564 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.941599 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.941839 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.941935 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:14:28.942076 8367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:28.942250 8367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81832955c5844feba40197b88c2f9511, State: Initialized, Role: FOLLOWER
I20260812 00:14:28.942422 8367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:28.942529 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.933282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.16.99.254:34891)
0812 00:14:28.933321 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.933323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.933330 (+ 7us) server_negotiation.cc:407] Connection header received
0812 00:14:28.933363 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.933363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.933372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.933416 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.934010 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.934544 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.935442 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.935646 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.935730 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.938970 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.938971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.938971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.938979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.940888 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.941358 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.941359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.941360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.941415 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.941652 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.941653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.941653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.941805 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.941977 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.942030 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.942106 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:14:28.942667 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.932806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.253:38051 (local address 127.0.0.1:35724)
0812 00:14:28.933037 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.933049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.933096 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.933286 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.933287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.933451 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.933499 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.933501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.934528 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.934529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.935400 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.935403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.935978 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.935979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.936036 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.936436 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.936444 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.938682 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.940830 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.940831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.940831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.941054 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.941276 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.941277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.941277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.941340 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.941669 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.941670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.941717 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.942285 (+ 568us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.942434 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260812 00:14:28.942842 8367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:28.942967 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:28.943061 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:28.943120 8522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } has no permanent_uuid. Determining permanent_uuid...
W20260812 00:14:28.944989 8540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:28.945084 8540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.252@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.99.253@KRBTEST.COM
I20260812 00:14:28.945855 8296 master_runner.cc:186] Error getting master registration for 127.16.99.253:38051: OK, instance_id { permanent_uuid: "6ceb183b90f5458190d12681084bc595" instance_seqno: 1786493668739544 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:14:28.949407 8296 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:14:28.953089 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.943385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57676)
0812 00:14:28.943424 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.943434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.943470 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.943741 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.943742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.943922 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.943965 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.943967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.945119 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.945120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.945918 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.945921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.946535 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.946536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.946589 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.947012 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.947018 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.949554 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.951770 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.951771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.951772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.952013 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.952247 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.952248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.952249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.952309 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.952717 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.952718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.952758 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.952807 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.952943 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 00:14:28.953472 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.943221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.16.99.253:38051)
0812 00:14:28.943258 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.943260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.943450 (+ 190us) server_negotiation.cc:407] Connection header received
0812 00:14:28.943575 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.943576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.943588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.943641 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.944222 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.944902 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.945739 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.945950 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.946033 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.950203 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.950204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.950205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.950213 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.952134 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.952553 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.952555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.952555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.952614 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.952865 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.952866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.952867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.953008 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.953190 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.953271 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.953361 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 00:14:28.953612 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.943412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.16.99.252:40389)
0812 00:14:28.943607 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.943609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.943619 (+ 10us) server_negotiation.cc:407] Connection header received
0812 00:14:28.943643 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.943644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.943655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.944041 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.944069 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.944979 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.946050 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.946446 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.946747 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.949665 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.949667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.949667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.949677 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.951671 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.952094 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.952095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.952096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.952159 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.952393 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.952394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.952394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.952632 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.953302 (+ 670us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.953343 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.953514 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260812 00:14:28.954862 16783 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 00:14:28.955240 8522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: Bootstrap starting.
I20260812 00:14:28.955852 8340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "81832955c5844feba40197b88c2f9511"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550
I20260812 00:14:28.956002 8522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:28.956022 8340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 00:14:28.956091 8340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:28.956408 8522 log.cc:824] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:28.957496 8522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: No bootstrap required, opened a new log
I20260812 00:14:28.957567 8340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:28.957911 8340 leader_election.cc:288] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [CANDIDATE]: Term 1 election: Requested vote from peers 6ceb183b90f5458190d12681084bc595 (127.16.99.253:38051), e8cc029345b2417d9e10b041f83227d9 (127.16.99.252:40389)
I20260812 00:14:28.960270 8522 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:28.960518 8522 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8cc029345b2417d9e10b041f83227d9, State: Initialized, Role: FOLLOWER
I20260812 00:14:28.960637 8522 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:28.961017 8522 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:28.961091 8495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81832955c5844feba40197b88c2f9511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e8cc029345b2417d9e10b041f83227d9"
I20260812 00:14:28.961221 8495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:28.962632 8495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81832955c5844feba40197b88c2f9511 in term 1.
I20260812 00:14:28.962915 8313 leader_election.cc:302] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81832955c5844feba40197b88c2f9511, e8cc029345b2417d9e10b041f83227d9; no voters:
I20260812 00:14:28.963089 8529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 00:14:28.963248 8529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [term 1 LEADER]: Becoming Leader. State: Replica: 81832955c5844feba40197b88c2f9511, State: Running, Role: LEADER
I20260812 00:14:28.963341 8529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:28.963783 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81832955c5844feba40197b88c2f9511. Latest consensus state: current_term: 1 leader_uuid: "81832955c5844feba40197b88c2f9511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:28.963853 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:28.964097 8565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260812 00:14:28.963591 8561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:28.964401 8561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:28.963722 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:28.964489 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:28.964345 8565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 00:14:28.966218 8495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [term 1 FOLLOWER]: Refusing update from remote peer 81832955c5844feba40197b88c2f9511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:28.966647 8529 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:28.966361 8451 master_runner.cc:186] Error getting master registration for 127.16.99.253:38051: OK, instance_id { permanent_uuid: "6ceb183b90f5458190d12681084bc595" instance_seqno: 1786493668739544 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 00:14:28.968806 8451 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 00:14:28.969898 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81832955c5844feba40197b88c2f9511. Latest consensus state: current_term: 1 leader_uuid: "81832955c5844feba40197b88c2f9511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:28.970007 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:28.971156 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.959824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.16.99.254:34891)
0812 00:14:28.959870 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.959873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.960412 (+ 539us) server_negotiation.cc:407] Connection header received
0812 00:14:28.960558 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.960559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.960571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.960606 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.961394 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.962260 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.963454 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.963790 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.963841 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.966983 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.966985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.966985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.966993 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.969278 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.969786 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.969788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.969789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.969869 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.970116 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.970118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.970118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.970569 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.970673 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.970935 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.971031 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 00:14:28.973223 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81832955c5844feba40197b88c2f9511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:28.973320 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: This master's current role is: LEADER
I20260812 00:14:28.973606 8566 mvcc.cc:205] Tried to move back new op lower bound from 7317478068083298304 to 7317478068074622976. Current Snapshot: MvccSnapshot[applied={T|T < 7317478068083298304}]
I20260812 00:14:28.973974 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81832955c5844feba40197b88c2f9511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:28.974061 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:28.974655 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.959492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.16.99.253:38051)
0812 00:14:28.959536 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.959539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.959811 (+ 272us) server_negotiation.cc:407] Connection header received
0812 00:14:28.960004 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.960005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.960016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.960047 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.965606 (+ 5559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.966329 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.967983 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.968426 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.968509 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.970044 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.970045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.970046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.970053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.971690 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.972483 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.972484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.972485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.972573 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.972893 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.972895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.972895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.973047 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.973164 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.974460 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.974556 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 00:14:28.975574 8565 catalog_manager.cc:1347] Generated new cluster ID: f2ef6ad0c1e345eb8bb281adecdf8af9
I20260812 00:14:28.975637 8565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 00:14:28.979266 8444 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } attempt: 3
I20260812 00:14:28.983738 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.960138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.16.99.252:40389)
0812 00:14:28.962514 (+ 2376us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.962516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.971575 (+ 9059us) server_negotiation.cc:407] Connection header received
0812 00:14:28.971708 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.971709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.971722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.971781 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.976116 (+ 4335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.976891 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.977994 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.978313 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.978401 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.980198 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.980200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.980200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.980208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.982214 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.982728 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.982729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.982730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.982798 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.983077 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.983078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.983078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.983219 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.983330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.983540 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.983638 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2355}
Aug 12 00:14:28 dist-test-slave-zpfg krb5kdc[8283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786493668, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.99.253@KRBTEST.COM for kudu/127.16.99.252@KRBTEST.COM
I20260812 00:14:28.991479 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.979828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.99.252:40389 (local address 127.0.0.1:57694)
0812 00:14:28.979895 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 00:14:28.979907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 00:14:28.979960 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 00:14:28.980325 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 00:14:28.980326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 00:14:28.980523 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 00:14:28.980580 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.980582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.982141 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.982142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.983012 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 00:14:28.983015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.983641 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.983658 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 00:14:28.983720 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.984243 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 00:14:28.984249 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 00:14:28.986533 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 00:14:28.990012 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.990014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.990014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.990255 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.990594 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 00:14:28.990594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 00:14:28.990595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 00:14:28.990681 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 00:14:28.991152 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 00:14:28.991153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 00:14:28.991215 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 00:14:28.991263 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 00:14:28.991370 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 00:14:28.992337 8565 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 00:14:28.992694 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:28.979877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.16.99.252:40389)
0812 00:14:28.979926 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:28.979929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:28.979941 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:14:28.980065 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:28.980067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:28.980096 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:28.980217 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:28.980779 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.981994 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.983140 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:28.983484 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:28.983553 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:28.986662 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 00:14:28.986663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 00:14:28.986664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 00:14:28.986691 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 00:14:28.989892 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.990387 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.990388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.990389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.990507 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 00:14:28.990808 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 00:14:28.990809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 00:14:28.990809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 00:14:28.991034 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 00:14:28.992103 (+ 1069us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:28.992218 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 00:14:28.992373 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 00:14:28.993110 8565 catalog_manager.cc:1504] Loading token signing keys...
I20260812 00:14:28.994750 8444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: Bootstrap starting.
I20260812 00:14:28.995546 8444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: Neither blocks nor log segments found. Creating new log.
I20260812 00:14:28.995882 8444 log.cc:824] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: Log is configured to *not* fsync() on all Append() calls
I20260812 00:14:28.998796 8444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: No bootstrap required, opened a new log
I20260812 00:14:29.000955 8444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:29.001114 8444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ceb183b90f5458190d12681084bc595, State: Initialized, Role: FOLLOWER
I20260812 00:14:29.001199 8444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } }
I20260812 00:14:29.002177 8416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 00:14:29.003249 8416 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [term 1 FOLLOWER]: Refusing update from remote peer 81832955c5844feba40197b88c2f9511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 00:14:29.003641 8562 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 00:14:29.002204 8417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "81832955c5844feba40197b88c2f9511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ceb183b90f5458190d12681084bc595"
I20260812 00:14:29.005544 8417 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81832955c5844feba40197b88c2f9511 in term 1.
I20260812 00:14:29.006453 8573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:29.006556 8573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:29.002065 8444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 00:14:29.008086 8565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511: Generated new TSK 0
I20260812 00:14:29.008296 8565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 00:14:29.009752 8573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81832955c5844feba40197b88c2f9511. Latest consensus state: current_term: 1 leader_uuid: "81832955c5844feba40197b88c2f9511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:29.009869 8573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:29.009925 8573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81832955c5844feba40197b88c2f9511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:29.009965 8573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 00:14:29.015024 8575 mvcc.cc:205] Tried to move back new op lower bound from 7317478068186820608 to 7317478068074622976. Current Snapshot: MvccSnapshot[applied={T|T < 7317478068083298304}]
W20260812 00:14:29.015812 8587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 00:14:29.015877 8587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 00:14:29.019433 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81832955c5844feba40197b88c2f9511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81832955c5844feba40197b88c2f9511" member_type: VOTER last_known_addr { host: "127.16.99.254" port: 34891 } } peers { permanent_uuid: "6ceb183b90f5458190d12681084bc595" member_type: VOTER last_known_addr { host: "127.16.99.253" port: 38051 } } peers { permanent_uuid: "e8cc029345b2417d9e10b041f83227d9" member_type: VOTER last_known_addr { host: "127.16.99.252" port: 40389 } } }
I20260812 00:14:29.019559 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81832955c5844feba40197b88c2f9511 [sys.catalog]: This master's current role is: LEADER
W20260812 00:14:29.038697 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.036915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.16.99.253:38051)
0812 00:14:29.036962 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.036964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.037918 (+ 954us) server_negotiation.cc:407] Connection header received
0812 00:14:29.038029 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.038030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.038042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.038075 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.038616 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35740: BlockingRecv error: recv got EOF from 127.0.0.1:35740 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 00:14:29.039649 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.037380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.16.99.254:34891)
0812 00:14:29.037427 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.037429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.039119 (+ 1690us) server_negotiation.cc:407] Connection header received
0812 00:14:29.039217 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.039218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.039231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.039259 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.039569 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35578: BlockingRecv error: recv got EOF from 127.0.0.1:35578 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:14:29.040501 8525 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.037732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.16.99.252:40389)
0812 00:14:29.037780 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.037782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.039977 (+ 2195us) server_negotiation.cc:407] Connection header received
0812 00:14:29.040102 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.040103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.040115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.040144 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.040442 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57700: BlockingRecv error: recv got EOF from 127.0.0.1:57700 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260812 00:14:29.043335 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.042473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.16.99.253:38051)
0812 00:14:29.042511 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.042513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.042786 (+ 273us) server_negotiation.cc:407] Connection header received
0812 00:14:29.042892 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.042893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.042914 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.042943 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.043285 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35744: BlockingRecv error: recv got EOF from 127.0.0.1:35744 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 00:14:29.046659 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.045773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.16.99.253:38051)
0812 00:14:29.045810 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.045812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.046071 (+ 259us) server_negotiation.cc:407] Connection header received
0812 00:14:29.046176 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.046177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.046188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.046219 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.046599 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35748: BlockingRecv error: recv got EOF from 127.0.0.1:35748 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 00:14:29.051003 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.049941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.16.99.253:38051)
0812 00:14:29.049993 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.049995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.050328 (+ 333us) server_negotiation.cc:407] Connection header received
0812 00:14:29.050488 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.050489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.050501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.050530 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.050949 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35758: BlockingRecv error: recv got EOF from 127.0.0.1:35758 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260812 00:14:29.057246 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.056314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.16.99.253:38051)
0812 00:14:29.056357 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.056359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.056676 (+ 317us) server_negotiation.cc:407] Connection header received
0812 00:14:29.056781 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.056782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.056794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.056824 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.057192 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35762: BlockingRecv error: recv got EOF from 127.0.0.1:35762 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:14:29.067675 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.066684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35774 (local address 127.16.99.253:38051)
0812 00:14:29.066731 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.066733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.067034 (+ 301us) server_negotiation.cc:407] Connection header received
0812 00:14:29.067148 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.067149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.067162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.067194 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.067617 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35774: BlockingRecv error: recv got EOF from 127.0.0.1:35774 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260812 00:14:29.086549 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.085390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.16.99.253:38051)
0812 00:14:29.085439 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.085441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.085826 (+ 385us) server_negotiation.cc:407] Connection header received
0812 00:14:29.085950 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.085952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.085966 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.085999 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.086486 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35784: BlockingRecv error: recv got EOF from 127.0.0.1:35784 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 00:14:29.121413 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.120247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.16.99.253:38051)
0812 00:14:29.120308 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.120310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.120664 (+ 354us) server_negotiation.cc:407] Connection header received
0812 00:14:29.120791 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.120793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.120806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.120862 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.121347 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35786: BlockingRecv error: recv got EOF from 127.0.0.1:35786 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260812 00:14:29.187920 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.187078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.16.99.253:38051)
0812 00:14:29.187163 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.187165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.187305 (+ 140us) server_negotiation.cc:407] Connection header received
0812 00:14:29.187355 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.187356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.187367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.187423 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.187857 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35798: BlockingRecv error: recv got EOF from 127.0.0.1:35798 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 00:14:29.317975 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.317122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.16.99.253:38051)
0812 00:14:29.317184 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.317186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.317291 (+ 105us) server_negotiation.cc:407] Connection header received
0812 00:14:29.317363 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.317364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.317375 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.317421 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.317890 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35802: BlockingRecv error: recv got EOF from 127.0.0.1:35802 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260812 00:14:29.576303 8448 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 00:14:29.575626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.16.99.253:38051)
0812 00:14:29.575701 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:29.575703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:29.575715 (+ 12us) server_negotiation.cc:407] Connection header received
0812 00:14:29.575742 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:29.575743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:29.575753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:29.575822 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 00:14:29.576250 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35804: BlockingRecv error: recv got EOF from 127.0.0.1:35804 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 00:14:29.964968 8561 catalog_manager.cc:1259] Loaded cluster ID: f2ef6ad0c1e345eb8bb281adecdf8af9
I20260812 00:14:29.965091 8561 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: loading cluster ID for follower catalog manager: success
I20260812 00:14:29.965982 8561 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: acquiring CA information for follower catalog manager: success
I20260812 00:14:29.966454 8561 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8cc029345b2417d9e10b041f83227d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:14:30.016342 8587 catalog_manager.cc:1259] Loaded cluster ID: f2ef6ad0c1e345eb8bb281adecdf8af9
I20260812 00:14:30.016430 8587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: loading cluster ID for follower catalog manager: success
I20260812 00:14:30.017318 8587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: acquiring CA information for follower catalog manager: success
I20260812 00:14:30.017683 8587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ceb183b90f5458190d12681084bc595: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 00:14:30.093112 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.089615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.16.99.253:38051)
0812 00:14:30.089753 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:30.089755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:30.089855 (+ 100us) server_negotiation.cc:407] Connection header received
0812 00:14:30.089921 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:30.089922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:30.089976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:30.090043 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:30.090460 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.090997 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.092075 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.092343 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.092448 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:30.092664 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:30.092839 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:30.092938 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:30.092985 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 00:14:30.093009 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260812 00:14:30.097854 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.095336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.16.99.252:40389)
0812 00:14:30.095442 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:30.095444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:30.095450 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:14:30.095490 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:30.095490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:30.095528 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:30.095614 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:30.095748 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.096270 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.097028 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.097234 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.097309 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:30.097493 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:30.097603 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:30.097686 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:30.097728 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 00:14:30.097759 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260812 00:14:30.102587 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.099637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.16.99.253:38051)
0812 00:14:30.099693 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:30.099695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:30.099798 (+ 103us) server_negotiation.cc:407] Connection header received
0812 00:14:30.099911 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:30.099912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:30.099924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:30.099954 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:30.100129 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.100693 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.101740 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.102060 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.102126 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:30.102267 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:30.102300 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:30.102378 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:30.102482 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 00:14:30.102513 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 00:14:30.102798 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.100024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.16.99.252:40389)
0812 00:14:30.100062 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:30.100064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:30.100070 (+ 6us) server_negotiation.cc:407] Connection header received
0812 00:14:30.100091 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:30.100092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:30.100103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:30.100150 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:30.100325 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.101061 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.101903 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.102213 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.102272 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:30.102527 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:30.102562 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:30.102618 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:30.102684 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 00:14:30.102706 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 00:14:30.103134 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 00:14:30.099737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.16.99.254:34891)
0812 00:14:30.099870 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 00:14:30.099872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 00:14:30.099881 (+ 9us) server_negotiation.cc:407] Connection header received
0812 00:14:30.099926 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 00:14:30.099927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 00:14:30.099982 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 00:14:30.100065 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 00:14:30.100246 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.101035 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.102051 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 00:14:30.102381 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 00:14:30.102443 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 00:14:30.102641 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 00:14:30.102757 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 00:14:30.102817 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 00:14:30.102995 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 00:14:30.103030 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260812 00:14:30.106719 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 8296
I20260812 00:14:30.113940 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 8374
I20260812 00:14:30.120646 16783 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcVEJCj/build/release/bin/kudu with pid 8451
2026-08-12T00:14:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3005 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6073 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288522 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST